This project is mirrored from https://gitee.com/yxydde/incubator-kyuubi.git.
Pull mirroring failed .
Repository mirroring has been paused due to too many failed attempts. It can be resumed by a project maintainer.
Repository mirroring has been paused due to too many failed attempts. It can be resumed by a project maintainer.
- 11 Mar, 2022 1 commit
-
-
SteNicholas authored
### _Why are the changes needed?_ We should relocate fb303 classes to avoid conflict, fix #2093. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2096 from SteNicholas/KYUUBI-2093. Closes #1948 2095f835 [SteNicholas] [KYUUBI #2093] kyuubi beeline can not connect the kyuubi server due to exception Authored-by:
SteNicholas <programgeek@163.com> Signed-off-by:
Cheng Pan <chengpan@apache.org> (cherry picked from commit 12d56422 ) Signed-off-by:
-
- 10 Mar, 2022 3 commits
-
-
SteNicholas authored
### _Why are the changes needed?_ `logCaptureThread` does not catch sparksubmit exception. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2090 from SteNicholas/KYUUBI-2078. Closes #2078 11ff6824 [SteNicholas] [KYUUBI #2078] logCaptureThread does not catch sparksubmit exception Authored-by:
Kent Yao <yao@apache.org> (cherry picked from commit cf014ee2 ) Signed-off-by:
-
Tianlin Liao authored
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> To close #1936 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2059 from lightning-L/kyuubi-1936. Closes #1936 e9c83ef6 [Tianlin Liao] [KYUUBI #1936] send credentials when opening session and wait for completion Authored-by:
Tianlin Liao <tiliao@ebay.com> Signed-off-by:
ulysses-you <ulyssesyou@apache.org> (cherry picked from commit 8e983a19 ) Signed-off-by:
-
yanghua authored
… engine <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2080 from yanghua/KYUUBI-2079. Closes #2079 2bf98720 [yanghua] [KYUUBI #2079] Update kyuubi layer source file to add flink and trino engine Authored-by:
yanghua <yanghua1127@gmail.com> Signed-off-by:
-
- 09 Mar, 2022 2 commits
-
-
Fei Wang authored
### _Why are the changes needed?_ We met below issue when fetching result. ``` Caused by: java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException:46 7022 at sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:453) 7023 at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2397) 7024 at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312) 7025 at java.util.Calendar.setTimeInMillis(Calendar.java:1804) 7026 at java.util.Calendar.setTime(Calendar.java:1770) 7027 at java.text.SimpleDateFormat.format(SimpleDateFormat.java:943) 7028 at java.text.SimpleDateFormat.format(SimpleDateFormat.java:936) 7029 at java.text.DateFormat.format(DateFormat.java:345) 7030 at org.apache.kyuubi.schema.RowSet$.toHiveString(RowSet.scala:245) 7031 at org.apache.kyuubi.schema.RowSet$.$anonfun$toTColumn$3(RowSet.scala:120) 7032 at scala.collection.immutable.List.map(List.scala:290) 7033 at org.apache.kyuubi.schema.RowSet$.toTColumn(RowSet.scala:115) 7034 at org.apache.kyuubi.schema.RowSet$.$anonfun$toColumnBasedSet$1(RowSet.scala:65) 7035 at org.apache.kyuubi.schema.RowSet$.$anonfun$toColumnBasedSet$1$adapted(RowSet.scala:64) 7036 at scala.collection.immutable.List.foreach(List.scala:392) 7037 at org.apache.kyuubi.schema.RowSet$.toColumnBasedSet(RowSet.scala:64) 7038 at org.apache.kyuubi.schema.RowSet$.toTRowSet(RowSet.scala:47) 7039 at org.apache.kyuubi.engine.spark.operation.SparkOperation.getNextRowSet(SparkOperation.scala:183) 7040 at org.apache.kyuubi.operation.OperationManager.getOperationNextRowSet(OperationManager.scala:116) 7041 at org.apache.kyuubi.session.AbstractSession.fetchResults(AbstractSession.scala:197) 7042 at org.apache.kyuubi.service.AbstractBackendService.fetchResults(AbstractBackendService.scala:169) 7043 at org.apache.kyuubi.service.ThriftBinaryFrontendService.FetchResults(ThriftBinaryFrontendService.scala:505) ``` The root cause is that the date time formatter used to convert the result to rowSet is not thread-safe. In this pr, we use thread-safe FastDateFormat instead of SimpleDateFormat. ### _How was this patch tested?_ Existing UT. Closes #2075 from turboFei/thread_safe_fat_stimeformatter. Closes #2075 44ae8fd1 [Fei Wang] Using thread-safe FastDateFormat instead of SimpleDateFormat Authored-by:
Fei Wang <fwang12@ebay.com> Signed-off-by:
-
yanghua authored
### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2064 from yanghua/KYUUBI-2063. Closes #2063 35ea82cf [yanghua] [KYUUBI #2063] Fix engine idle timeout lose efficacy for Flink Engine Authored-by:
-
- 08 Mar, 2022 4 commits
-
-
Cheng Pan authored
### _Why are the changes needed?_ Document incremental collection, close #1215 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate <img width="1914" alt="1" src="https://user-images.githubusercontent.com/26535726/157191008-451bee2a-eb6b-4bb6-869b-5b0f75a21448.png"> <img width="1919" alt="2" src="https://user-images.githubusercontent.com/26535726/157191016-e183bbf5-aa4a-496d-a250-5d14cf04101d.png"> <img width="1920" alt="3" src="https://user-images.githubusercontent.com/26535726/157191026-343a39d7-0ab8-4886-9a51-3670394ef6be.png"> - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2057 from pan3793/doc. Closes #1215 82677a03 [Cheng Pan] grammar 3be0e3f1 [Cheng Pan] fix b467e975 [Cheng Pan] Update d256cbaf [Cheng Pan] compress picture b3c5fb64 [Cheng Pan] Fix 4c06307c [Cheng Pan] [KYUUBI #1215][DOC] Document incremental collection Authored-by:
-
--replace-all authored
### _Why are the changes needed?_ Close #2043 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate  - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2046 from RishiKumarRay/master. Closes #2043 bf9a4a0b [--replace-all] fixed sql issue c119e5c2 [Rishi Kumar Ray] Merge branch 'apache:master' into master 323d81c7 [--replace-all] Upgrade log4j/2.x/ to 2.17.2 Lead-authored-by:
--replace-all <rishi.ray@knoldus.com> Co-authored-by:
Rishi Kumar Ray <87641376+RishiKumarRay@users.noreply.github.com> Signed-off-by:
-
Cheng Pan authored
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> We need to clear job group information for init SQL, otherwise in incremental collection mode, the fetchNext will always display wrong description. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate Before this patch: <img width="1916" alt="incremental_collect" src="https://user-images.githubusercontent.com/26535726/157165964-502ad809-9d99-44de-b08c-131d135ce33d.png"> After this patch: <img width="1917" alt="Xnip2022-03-08_12-04-50" src="https://user-images.githubusercontent.com/26535726/157165975-68eddf24-e0ad-4db2-931c-c75541802a7b.png"> - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2060 from pan3793/clear. Closes #2060 76a67372 [Cheng Pan] Clear job group for init SQL Authored-by:
-
SteNicholas authored
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Support closing Flink SQL engine process. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2054 from SteNicholas/KYUUBI-1819. Closes #2054 992bd0b2 [SteNicholas] [KYUUBI-1819] Support closing Flink SQL engine process 69567120 [SteNicholas] [KYUUBI-1819] Support closing Flink SQL engine process Authored-by:
-
- 07 Mar, 2022 2 commits
-
-
jiaoqingbo authored
### _Why are the changes needed?_ fix #2055 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2056 from jiaoqingbo/2055. Closes #2055 52a2e623 [jiaoqingbo] [KYUUBI #2055] correct the log service name Authored-by:
jiaoqingbo <1178404354@qq.com> Signed-off-by:
-
sychen authored
### _Why are the changes needed?_ Support more MySQL JDBC driver versions. The user may use the 5.x mysql driver to connect to the kyuubi server. https://github.com/apache/incubator-kyuubi/issues/2047 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2048 from cxzl25/KYUUBI-2047. Closes #2047 03cd15f2 [sychen] Support more MySQL JDBC driver versions Authored-by:
sychen <sychen@trip.com> Signed-off-by:
-
- 05 Mar, 2022 6 commits
-
-
Kent Yao authored
-
Kent Yao authored
-
Kent Yao authored
-
jiaoqingbo authored
### _Why are the changes needed?_ Fix #2012 #1180 In `EventLoggingService`, when `EventLoggerType` is set to `JSON` Kyuubi creates an error directory, it should not be the `ENGINE_EVENT_JSON_LOG_PATH` but the `SERVER_EVENT_JSON_LOG_PATH`. <img width="557" alt="微信图片_20220304165614" src="https://user-images.githubusercontent.com/14961757/156771039-e18fcdbe-8b99-4190-9b14-6953daa55e5f.png"> <img width="605" alt="微信图片_20220304165633" src="https://user-images.githubusercontent.com/14961757/156771053-defdee62-67f9-4f16-9d4a-8ef170304221.png"> rename `EventLoggingService` in `kyuubi-server` to `KyuubiEventLoggingService` rename `EventLoggingService` in `kyuubi-spark-sql-engine` to `SparkEventLoggingService` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2037 from jiaoqingbo/kyuubi2012. Closes #2012 714e5ea5 [jiaoqingbo] fix ut failed 10d1a151 [jiaoqingbo] execute dev/format cb99a458 [jiaoqingbo] [KYUUBI #2012] rename EventLoggingService and creates an correct directory Authored-by:
-
Jeff Zhang authored
### _Why are the changes needed?_ Update `.asf.yaml` to redirect Issues/PR Notifications to notificationskyuubi.apache.org ### _How was this patch tested?_ No test needed Closes #2042 from zjffdu/notification. Closes #2042 1f16cb4a [Jeff Zhang] Redirect Issues/PR Notifications to notifications@kyuubi.apache.org Authored-by:
Jeff Zhang <zjffdu@apache.org> Signed-off-by:
-
Cheng Pan authored
### _Why are the changes needed?_ Replaced by FlinkOperationSuite ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2039 from pan3793/flink. Closes #2039 f9fd6674 [Cheng Pan] Remove LegacyFlinkOperationSuite Authored-by:
-
- 04 Mar, 2022 3 commits
-
-
SteNicholas authored
### _Why are the changes needed?_ Upgrade `hive-service-rpc` to 3.1.2. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2031 from SteNicholas/KYUUBI-1961. Closes #1961 88c29557 [SteNicholas] [KYUUBI #1961] Upgrade hive-service-rpc to 3.1.2 Authored-by:
-
sychen authored
### _Why are the changes needed?_ Introduced engine type and subdomain, kyuubi-ctl does not support managing engine. Add two engine-related parameters(--engine-type --engine-subdomain). https://github.com/apache/incubator-kyuubi/issues/2008 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2009 from cxzl25/KYUUBI-2008. Closes #2008 213a8d9e [sychen] remove println c3895b2e [sychen] fix style fa1e8991 [sychen] fix ut 302ae757 [sychen] support engine type and subdomain Authored-by:
-
sychen authored
### _Why are the changes needed?_ When the get server and list server commands are displayed, when the namespace exceeds the minimum width, it will be glued to the hostname. Add a space beyond the minimum cell width in non-verbose mode. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [x] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2007 from cxzl25/KYUUBI-2006. Closes #2006 68bacef3 [sychen] Add a space beyond the minimum cell width in non-verbose mode Authored-by:
-
- 03 Mar, 2022 6 commits
-
-
HYBG-1126 authored
cretate ns kyuubi ---> create ns kyuubi ### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1999 from HYBG-1126/patch-1. Closes #1999 f68f73e9 [HYBG] Update quick_start_with_helm.md Lead-authored-by:
HYBG-1126 <hybg1126@outlook.com> Co-authored-by:
HYBG <44338121+HYBG-1126@users.noreply.github.com> Signed-off-by:
-
Ada Wong authored
### _Why are the changes needed?_ Hadoop 3.3.2 is out. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2003 from deadwind4/KYUUBI-2000. Closes #2000 3b5b3f85 [Ada Wong] [KYUUBI #2000][DEPS] Bump Hadoop 3.3.2 Authored-by:
Ada Wong <rsl4@foxmail.com> Signed-off-by:
-
Kent Yao authored
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> Add CONTRIBUTING.md to list COMMUNITY resources that I contributor can use delete contributions and improvement-proposal in main repo ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #2004 from yaooqinn/CONTRIBUTING. Closes #2004 ddbcd521 [Kent Yao] CONTRIBUTING f81c14b3 [Kent Yao] Merge branch 'master' of github.com:apache/incubator-kyuubi into CONTRIBUTING 3bdbdc9b [Kent Yao] CONTRIBUTING 574ceff2 [Kent Yao] CONTRIBUTING f0a6b4c4 [Kent Yao] CONTRIBUTING c55c6a02 [Kent Yao] CONTRIBUTING Authored-by:
-
Cheng Pan authored
### _Why are the changes needed?_ [THRIFT-4805](https://issues.apache.org/jira/browse/THRIFT-4805) has been fixed in thrift 0.13.0, we can remove the workaround of log4j2 configurations because we use thrift 0.16.0 now, the change also removes the following warnings. ``` 2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" 2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" ``` `maven-surefire-plugin` is only used in `kyuubi-hive-jdbc`, we should disable it in other modules. ``` [INFO] --- maven-surefire-plugin:2.22.0:test (default-test) kyuubi-ctl_2.12 --- [INFO] [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- 2022-03-01 23:18:16,530 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" 2022-03-01 23:18:16,543 main ERROR Filters contains invalid attributes "onMatch", "onMismatch" [INFO] Running org.apache.kyuubi.ctl.ServiceControlCliSuite [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 s - in org.apache.kyuubi.ctl.ServiceControlCliSuite [INFO] Running org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.kyuubi.ctl.ServiceControlCliArgumentsSuite [INFO] [INFO] Results: [INFO] [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1996 from pan3793/log. Closes #1996 291441c0 [Cheng Pan] nit f2c94246 [Cheng Pan] Tune surefire plugin in conf e1ee6402 [Cheng Pan] Clean up log4j2 conf Authored-by:
-
Kent Yao authored
### _Why are the changes needed?_ # Apache Kyuubi(Incubating) Maturity Self Assessment ## Overview This is an assessment of the overall maturity of Apache Kyuubi(Incubating) as an Apache project, meant to inform the decision (of mentors, community, incubator PMCs, and ASF Board of Directors) to graduate Kyuubi as a top-level Apache project. It is based on the [Apache Project Maturity Model](https://community.apache.org/apache-way/apache-project-maturity-model.html). Mentors and community members are encouraged to contribute to this page and comment on it, the following tables summarize our self-assessment against the **Apache Project Maturity Model**.
✅ means that the related item is in good status.⬜ means that the related item need long-term attention.❌ means that the related item need to be fixed ASAP. ## Code | ID | Description | Status | |-----|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | C10 | The project produces Open Source software for distribution to the public, at no charge. |✅ The project source code is licensed under the [Apache License, v2.0](https://github.com/apache/incubator-kyuubi/blob/master/LICENSE) | | C20 | Anyone can easily discover and access the project's code. |❌ The source code is available on GitHub directly,‼ Missing link on website | | C30 | Anyone using standard, widely-available tools, can build the code in a reproducible way. |❌ The source code is automatically built for each Pull Request commit, with license checks, style checks, doc builds unit and end to end tests, via GitHub actions and Travis. Development environment setup and build instructions to do the same locally are documented on the [website](https://kyuubi.apache.org/developer-tools.html).‼ The side is not sync with codebase. | | C40 | The full history of the project's code is available via a source code control system, in a way that allows anyone to recreate any released version. |✅ The project uses git to manage source code, documentation and website, all releases are tagged. The commit history is available from the beginning of the project. | | C50 | The source code control system establishes the provenance of each line of code in a reliable way, based on strong authentication of the committer. When third parties contribute code, commit messages provide reliable information about the code provenance. |✅ The project uses GitHub which managed by Apache Infra, it ensuring provenance of each line of code to a committer. All code is checked in after review and approval via GitHub pull requests. | ## Licenses and Copyright | ID | Description | Status | |------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | LC10 | The Apache License, version 2.0, covers the released code. |✅ The [LICENSE file](https://github.com/apache/incubator-kyuubi/blob/master/LICENSE) is in GitHub repository. [All releases](https://kyuubi.apache.org/releases.html) are verified by IPMC to contain the copy. | | LC20 | Libraries that are mandatory dependencies of the project's code do not create more restrictions than the Apache License does. |✅ [The list of mandatory dependencies](https://github.com/apache/incubator-kyuubi/tree/master/licenses-binary) have been reviewed to contain approved and compatible licenses only. | | LC30 | The libraries mentioned in LC20 are available as Open Source software. |✅ All mandatory dependencies listed in L20 are available as open source software. | | LC40 | Committers are bound by an Individual Contributor Agreement (the "Apache iCLA") that defines which code they may commit and how they need to identify code that is not their own. |✅ All committers have iCLAs on file before they have an apache account. | | LC50 | The project clearly defines and documents the copyright ownership of everything that the project produces. |✅ All files in the source repository have appropriate headers. [The Github workflow](https://github.com/apache/incubator-kyuubi/actions/workflows/license.yml) is in place to ensure every file has the expected license header. | ## Releases | ID | Description | Status | |------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | RE10 | Releases consist of source code, distributed using standard and open archive formats that are expected to stay readable in the long term. |✅ Recent releases are available via the [download page](https://kyuubi.apache.org/releases.html) on the website. All releases are available via the [apache archive](https://archive.apache.org/dist/incubator/kyuubi/). | | RE20 | The project's PMC (Project Management Committee, see CS10) approves each software release in order to make the release an act of the Foundation. |✅ All incubating releases have been approved by the Kyuubi community and the Incubator, all with at least 3 PPMC/IPMC votes and more +1 than -1. | | RE30 | Releases are signed and/or distributed along with digests that anyone can reliably use to validate the downloaded archives. |✅ [Releases](https://kyuubi.apache.org/releases.html) are signed, the [keys](https://downloads.apache.org/incubator/kyuubi/KEYS) are available from the download page. Every release has a corresponding hash(sha512) to prevent content tampering. | | RE40 | The project can distribute convenience binaries alongside source code, but they are not Apache Releases, they are provided with no guarantee. |✅ Convenience binaries are distributed alongside source code at the same time via <ul><li>[Maven Central Repository](https://mvnrepository.com/artifact/org.apache.kyuubi)</li><li>[DockerHub](https://hub.docker.com/r/apache/kyuubi)</li><li>dist.apache.org</li></ul> | | RE50 | The project documents a repeatable release process so that someone new to the project can independently generate the complete set of artifacts required for a release. |✅ [Release guide](https://github.com/apache/incubator-kyuubi/blob/master/docs/community/release.md) is available. The releases of the project have been performed by 4 different release managers. | ## Quality | ID | Description | Status | |------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| | QU10 | The project is open and honest about the quality of its code. Various levels of quality and maturity for various modules are natural and acceptable as long as they are clearly communicated. |✅ All issues are tracked on Kyuubi's GitHub [Issues](https://github.com/apache/incubator-kyuubi/issues). | | QU20 | The project puts a very high priority on producing secure software. |✅ Security issues are treated with the highest priority. We use Apache's default way to submit security related information. | | QU30 | The project provides a well-documented, secure and private channel to report security issues, along with a documented way of responding to them. |✅ Website provides a link to https://www.apache.org/security/ | | QU40 | The project puts a high priority on backwards compatibility and aims to document any incompatible changes and provide tools and documentation to help users transition to new features. |✅ Each release note contains all related issues and pull requests in the milestone, and extract mainly updates and API changes from milestones. | | QU50 | The project strives to respond to documented bug reports in a timely manner. |✅ The community is very active in responding to bug reports and usually fixes them within a short time period. | ## Community | ID | Description | Status | |------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | CO10 | The project has a well-known homepage that points to all the information required to operate according to this maturity model. |✅ The website describes of the project with download, user manual, technical details, how to contribute and team introduce. | | CO20 | The community welcomes contributions from anyone who acts in good faith and in a respectful manner, and who adds value to the project. |⬜ Committers are really welcome contributions and the community is actively seeking for new committers. The community has elected 0 new PPMC members and 2 new committers during incubation, based on meritocracy. | | CO30 | Contributions include source code, documentation, constructive bug reports, constructive discussions, marketing and generally anything that adds value to the project. |✅ The community welcomes all contributions. The contribution guide refers to non source code contribution. The community is very clear about contributions being more than code. | | CO40 | The community strives to be meritocratic and gives more rights and responsibilities to contributors who, over time, add value to the project. |⬜ . The community has elected 0 new PPMC members and 2 new committers during incubation, based on meritocracy. | | CO50 | The project documents how contributors can earn more rights such as commit access or decision power, and applies these principles consistently. |❌ ~The process to become a committer or PPMC member is documented on the website~. see, https://community.apache.org/newcommitter.html | | CO60 | The community operates based on consensus of its members (see CS10) who have decision power. Dictators, benevolent or not, are not welcome in Apache projects. |⬜ The project works to build consensus. All PPMC votes have been unanimous so far. | | CO70 | The project strives to answer user questions in a timely manner. |⬜ The project typically provides detailed answers to user questions within a few days via dev mailing list, GitHub issues and Github Discussions. | ## Consensus Building | ID | Description | Status | |------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | CS10 | The project maintains a public list of its contributors who have decision power. The project's PMC (Project Management Committee) consists of those contributors. |❌ ~Committers and PPMC members are listed on the website’s community page~ | | CS20 | Decisions require a consensus among PMC members and are documented on the project's main communications channel. The PMC takes community opinions into account, but the PMC has the final word. |✅ The project has been making important decisions on the mailing lists. | | CS30 | The project uses documented voting rules to build consensus when discussion is not sufficient. |✅ The project uses the standard ASF voting rules. | | CS40 | In Apache projects, vetoes are only valid for code commits. The person exercising the veto must justify it with a technical explanation, as per the Apache voting rules defined in CS30. |⬜ The project has not used a veto at any point during incubation. | | CS50 | All "important" discussions happen asynchronously in written form on the project's main communications channel. Offline, face-to-face or private discussions that affect the project are also documented on that channel. |⬜ The project has been making important decisions on the project mailing lists. Minor decisions may occasionally happen during code reviews, which are also asynchronous and in written form. | ## Independence | ID | Description | Status | |------|------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | IN10 | The project is independent from any corporate or organizational influence. |⬜ The project team gathers people from different companies (China Mobile, eBay, NetEase, T3). No company or organization has significantly more influence than any other. We can note a growth of the contributions coming from different contributors. | | IN20 | Contributors act as themselves, not as representatives of a corporation or organization. |✅ The contributors act on their own initiative without representing a corporation or organization. | ### _How was this patch tested?_ Pending reviews by Kyuubi Mentors, PPMC, and community. Closes #1986 from yaooqinn/MATURITY. Closes #1986 1c76752a [Kent Yao] Apache Kyuubi(Incubating) Maturity Self Assessment 3ce62adf [Kent Yao] Apache Kyuubi(Incubating) Maturity Self Assessment ac436ab3 [Kent Yao] Apache Kyuubi(Incubating) Maturity Self Assessment Authored-by: -
Cheng Pan authored
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> The `new SparkConf()` and `KyuubiConf()` executes early than UT, and can not re-initialize after changing system properties, which makes conf overwriting does not work. After the change, the `sparkConf.setIfMissing("spark.sql.catalogImplementation", defaultCat)` work as expected when we overwriting `spark.sql.catalogImplementation` in UT, and Hudi uses `in-memory` catalog to create table. <img width="891" alt="Xnip2022-03-02_19-10-06" src="https://user-images.githubusercontent.com/26535726/156351805-bb403ee8-0b89-4db2-b91f-d7b2b9957838.png"> ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1995 from pan3793/lazy. Closes #1988 c264a224 [Cheng Pan] Fix d1a23fbc [Cheng Pan] Lazy initialize vars depends on system level confs Authored-by:
-
- 02 Mar, 2022 1 commit
-
-
Ada Wong authored
### _Why are the changes needed?_ Users who use Flink or Trino need not set SPARK_HOME, so I remove SPARK_HOME predication in bin/kyuubi script. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1994 from deadwind4/KYUUBI-1992. Closes #1992 5f11df57 [Ada Wong] [KYUUBI #1992] Remove SPARK_HOME predication. Authored-by:
-
- 01 Mar, 2022 6 commits
-
-
wForget authored
[KYUUBI #1960] Skip creating sparksession and starting engine if the max initialization time exceeds <!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #1960 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [X] Add screenshots for manual tests if appropriate Test steps: 1. Start another application to fill the queue. 2. Modify the `stop-application.sh` so it doesn't work. 3. Submit a Kyuubi request. 4. After waiting for timeout, stop the application in step 1. We can get the following results: 1. The kyuubi request exits after timeout. 2. After obtaining resources, the orphaned engine app fails to execute and exits immediately.   - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #1966 from wForget/KYUUBI-1960. Closes #1960 0f977c85 [Wang Zhen] spotless 9bb2178c [Wang Zhen] Merge branch 'master' of https://github.com/apache/incubator-kyuubi into KYUUBI-1960 3effa4d1 [wForget] fix f9f13c6f [wForget] fix test abdc1f30 [wForget] fix 48b4bac3 [wForget] [KYUUBI-1960] Skip creating sparksession and starting engine if the max initialization time exceeds Lead-authored-by:
wForget <643348094@qq.com> Co-authored-by:
Wang Zhen <wangzhen07@qiyi.com> Signed-off-by:
-
Wang Zhen authored
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #1958 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1984 from wForget/KYUUBI-1958. Closes #1958 7702b23b [Wang Zhen] [KYUUBI-1958] Fix flaky test: statement redact for engine tab *** FAILED *** Authored-by:
-
Fu Chen authored
### _Why are the changes needed?_ After [PR-1807](https://github.com/apache/incubator-kyuubi/pull/1807), the error will still occur. ``` 11:11:05.589 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /Users/fchen/Project/bigdata/incubator-kyuubi/server_operation_logs/6a640069-6a26-4817-8ad7-77385360bb9d/a09d4b0a-85ec-4640-8ab9-c6a37146fd1f 11:11:05.603 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[a09d4b0a-85ec-4640-8ab9-c6a37146fd1f]: INITIALIZED_STATE -> PENDING_STATE, statement: LAUNCH_ENGINE 11:11:05.604 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[a09d4b0a-85ec-4640-8ab9-c6a37146fd1f]: PENDING_STATE -> RUNNING_STATE, statement: LAUNCH_ENGINE 11:11:25.075 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[a09d4b0a-85ec-4640-8ab9-c6a37146fd1f]: RUNNING_STATE -> FINISHED_STATE, statement: LAUNCH_ENGINE, time taken: 19.471 seconds 11:12:58.257 WARN org.apache.kyuubi.operation.KyuubiOperationManager: Operation OperationHandle [type=UNKNOWN_OPERATION, identifier: a09d4b0a-85ec-4640-8ab9-c6a37146fd1f] is timed-out and will be closed 11:12:58.258 INFO org.apache.kyuubi.operation.LaunchEngine: Processing anonymous's query[a09d4b0a-85ec-4640-8ab9-c6a37146fd1f]: FINISHED_STATE -> CLOSED_STATE, statement: LAUNCH_ENGINE 11:13:04.615 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [6a640069-6a26-4817-8ad7-77385360bb9d] 11:13:04.618 INFO org.apache.kyuubi.session.KyuubiSessionManager: SessionHandle [6a640069-6a26-4817-8ad7-77385360bb9d] is closed, current opening sessions 0 11:13:04.619 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error closing session: org.apache.kyuubi.KyuubiSQLException: Invalid OperationHandle [type=UNKNOWN_OPERATION, identifier: a09d4b0a-85ec-4640-8ab9-c6a37146fd1f] at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) ~[classes/:?] at org.apache.kyuubi.operation.OperationManager.removeOperation(OperationManager.scala:89) ~[classes/:?] at org.apache.kyuubi.operation.OperationManager.closeOperation(OperationManager.scala:104) ~[classes/:?] at org.apache.kyuubi.session.AbstractSession.$anonfun$closeOperation$1(AbstractSession.scala:181) ~[classes/:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:75) ~[classes/:?] at org.apache.kyuubi.session.AbstractSession.closeOperation(AbstractSession.scala:179) ~[classes/:?] at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:149) ~[classes/:?] at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:93) ~[classes/:?] at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49) ~[classes/:?] at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$closeSession(KyuubiServer.scala:142) ~[classes/:?] at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$closeSession$1(BackendServiceMetric.scala:43) ~[classes/:?] at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?] at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[classes/:?] at org.apache.kyuubi.server.BackendServiceMetric.closeSession(BackendServiceMetric.scala:43) ~[classes/:?] at org.apache.kyuubi.server.BackendServiceMetric.closeSession$(BackendServiceMetric.scala:41) ~[classes/:?] at org.apache.kyuubi.server.KyuubiServer$$anon$1.closeSession(KyuubiServer.scala:142) ~[classes/:?] at org.apache.kyuubi.service.TFrontendService.CloseSession(TFrontendService.scala:158) [classes/:?] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1397) [hive-service-rpc-2.3.9.jar:2.3.9] at org.apache.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1382) [hive-service-rpc-2.3.9.jar:2.3.9] at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) [libthrift-0.16.0.jar:0.16.0] at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) [libthrift-0.16.0.jar:0.16.0] at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) [classes/:?] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250) [libthrift-0.16.0.jar:0.16.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] 11:13:04.629 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [6a640069-6a26-4817-8ad7-77385360bb9d] ``` ``` A => LaunchEngine Operation TCloseSessionReq A-PENDING A-FINISHED | Δ Δ | | | ▽ ----t1------t2------t3------t4------------------------------t5------------t6--------------t7-> | | | ▽ | ▽ A-INITIALIZED ▽ A-CLOSED (timed-out) A-RUNNING |<-- operation.idle.timeout -->|<-- operation.idle.timeout -->| ``` 1. the variable `LaunchEngine.lastAccessTime` will be reset to `System.currentTimeMillis()` when the operation is timeouts(at time t5). https://github.com/apache/incubator-kyuubi/blob/67d1c5dd513327b350dac3d6855af5223a594087/kyuubi-common/src/main/scala/org/apache/kyuubi/operation/AbstractOperation.scala#L75-L89 2. when the Kyuubi server receives the request TCloseSessionReq at time t6 between t5 and t7. Since t6 < t7, `LaunchEngine.isTimedOut()` will return false, and we will close the operation LaunchEngine again. https://github.com/apache/incubator-kyuubi/blob/67d1c5dd513327b350dac3d6855af5223a594087/kyuubi-common/src/main/scala/org/apache/kyuubi/operation/AbstractOperation.scala#L189-L196 ### _How was this patch tested?_ Pass CI. Closes #1983 from cfmcgrady/kyuubi-1806-followup. Closes #1983 Closes #1806 c7642693 [Fu Chen] Fix LaunchEngine may be closed multiple times Authored-by:
Fu Chen <cfmcgrady@gmail.com> Signed-off-by:
-
Kent Yao authored
### _Why are the changes needed?_ ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1981 from yaooqinn/branch-dep. Closes #1981 07bf0e29 [Kent Yao] codec 96e41e7d [Kent Yao] codec 1da26432 [Kent Yao] guava 6cf0abb3 [Kent Yao] base64 & sha1 3b45ec2a [Kent Yao] base64 5490cfa7 [Kent Yao] http 6f4b48b8 [Kent Yao] n 2a87a4cf [Kent Yao] d a5ad3ccc [Kent Yao] Refine Deps in common module 24d5299e [Kent Yao] Refine Deps in common module Authored-by:
-
Wang Zhen authored
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> close #1972 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1982 from wForget/KYUUBI-1972. Closes #1972 2d83b245 [Wang Zhen] [KYUUBI-1972] Fix flaky test: SchedulerPoolSuite: Scheduler pool Authored-by:
-
Kent Yao authored
<!-- Thanks for sending a pull request! Here are some tips for you: 1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html 2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'. 3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'. --> ### _Why are the changes needed?_ <!-- Please clarify why the changes are needed. For instance, 1. If you add a feature, you can talk about the use case of it. 2. If you fix a bug, you can clarify why it is a bug. --> #1969 might be fixed by this. ``` 12:18:19.885 SparkTBinaryFrontendHandler-Pool: Thread-1260 INFO SparkSQLSessionManager: Session stopped due to shared level is Connection. 12:18:19.886 SparkTBinaryFrontendHandler-Pool: Thread-1260 INFO SparkSQLEngine: Service: [SparkTBinaryFrontend] is stopping. 12:18:19.886 SparkTBinaryFrontendHandler-Pool: Thread-1260 INFO SparkTBinaryFrontendService: SparkTBinaryFrontend has stopped 12:18:19.886 SparkTBinaryFrontendHandler-Pool: Thread-1260 INFO SparkTBinaryFrontendService: Service: [EngineServiceDiscovery] is stopping. 12:18:19.890 SparkTBinaryFrontendHandler-Pool: Thread-1260 DEBUG FailedDeleteManager: Path being added to guaranteed delete set: /kyuubi/CONNECTION/933147f8-1fcb-4cda-875f-0b219f97aaf3/serviceUri=localhost:43147;version=1.5.0-SNAPSHOT;sequence=0000000000 12:18:19.892 ProcessThread(sid:0 cport:40861): INFO PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x100000e6ac00000 type:delete cxid:0xb zxid:0x9 txntype:-1 reqpath:n/a Error Path:/kyuubi/CONNECTION/933147f8-1fcb-4cda-875f-0b219f97aaf3/serviceUri=localhost:43147;version=1.5.0-SNAPSHOT;sequence=0000000000 Error:KeeperErrorCode = NoNode for /kyuubi/CONNECTION/933147f8-1fcb-4cda-875f-0b219f97aaf3/serviceUri=localhost:43147;version=1.5.0-SNAPSHOT;sequence=0000000000 12:18:19.892 SyncThread:0 DEBUG FinalRequestProcessor: Processing request:: sessionid:0x100000e6ac00000 type:delete cxid:0xa zxid:0x8 txntype:2 reqpath:n/a 12:18:19.893 SparkTBinaryFrontendHandler-Pool: Thread-1260 ERROR ServiceDiscoveryClient: Failed to close the persistent ephemeral znodenull java.io.IOException: java.lang.InterruptedException at org.apache.curator.framework.recipes.nodes.PersistentNode.close(PersistentNode.java:296) ~[curator-recipes-2.12.0.jar:?] at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.deregisterService(ServiceDiscoveryClient.scala:117) ~[kyuubi-ha_2.12-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] at org.apache.kyuubi.ha.client.EngineServiceDiscovery.stop(EngineServiceDiscovery.scala:35) ~[kyuubi-ha_2.12-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] at org.apache.kyuubi.service.CompositeService.$anonfun$stop$2(CompositeService.scala:75) ~[kyuubi-common_2.12-1.5.0-SNAPSHOT.jar:1.5.0-SNAPSHOT] ``` the deregister for connection-level happens in session close and SparkTBinaryFrontendHandler pool, which might be stopped before finish executing zk node deletion. this PR contains some other race issues too ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1980 from yaooqinn/conc. Closes #1969 efd12707 [Kent Yao] address comments aed6636f [Kent Yao] Fix race on some service during stop phase 63592621 [Kent Yao] Fix race on some service during stop phase Authored-by:
-
- 28 Feb, 2022 1 commit
-
-
Kent Yao authored
### _Why are the changes needed?_ remove dead and duplicated codes for kyuubi events ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1979 from yaooqinn/branch-events. Closes #1979 213bccd5 [Kent Yao] Simplify Events Code 1f20a1c9 [Kent Yao] Simplify Events Code 97a04dcd [Kent Yao] Simplify Events Code f326bb0d [Kent Yao] Simplify Events Code 47c6b8df [Kent Yao] Simplify Events Code b18f19ba [Kent Yao] Simplify Events Code 788b9e4e [Kent Yao] Simplify Events Code Authored-by:
-
- 24 Feb, 2022 5 commits
-
-
Cheng Pan authored
### _Why are the changes needed?_ Use the default Spark version to run TPC-DS test. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1975 from pan3793/tpcds. Closes #1975 ecdda1d9 [Cheng Pan] style 1e8468f4 [Cheng Pan] nit 9666cd8b [Cheng Pan] Update golden file 10b38939 [Cheng Pan] default spark version c1c9b037 [Cheng Pan] [BUILD] Run TPC-DS test using Spark 3.2 Authored-by:
-
Cheng Pan authored
### _Why are the changes needed?_ This PR aims to support auto merge small files in multi insert statement, for example `FROM VALUES(1) INSERT INTO tmp1 SELECT * INSERT INTO tmp2 SELECT *;` will generate the following plan, `Union` is the root node instead of `InsertIntoHiveTable` ``` Union :- InsertIntoHiveTable : +- Project : +- LocalRelation +- InsertIntoHiveTable +- Project +- LocalRelation ``` This PR also fixed the `canInsertRepartitionByExpression`, previous it did not consider the `SubqueryAlias` which may cause inserting error `Repartition`/`Reblance` node and currupt the data distribution, e.g. `FROM (SELECT * FROM VALUES(1) DOSTRIBUTE BY col1) INSERT INTO tmp1 SELECT * INSERT INTO tmp2 SELECT *;` ``` Union :- InsertIntoHiveTable : +- Project : +- SubqueryAlias : +- RepartitionByExpression : +- Project : +- LocalRelation +- InsertIntoHiveTable +- Project +- SubqueryAlias +- RepartitionByExpression +- Project +- LocalRelation ``` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1974 from pan3793/ext. Closes #1974 56cd7734 [Cheng Pan] nit e0155c27 [Cheng Pan] Support merge small files in multi table insertion Authored-by: -
SteNicholas authored
### _Why are the changes needed?_ Make plan only mode skippable for configuable plans. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1967 from SteNicholas/KYUUBI-1963. Closes #1963 e253aec6 [SteNicholas] [KYUUBI #1963] Make plan only mode skippable for configuable plans b5cff405 [SteNicholas] [KYUUBI #1963] Make plan only mode skippable for configuable plans Authored-by:
-
Kent Yao authored
### _Why are the changes needed?_ Replace `mirror-cn` profile with Apache officially suggested dlcdn. mirror-cn is only suite for CN, while dlcdn is much wiser. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests ) locally before make a pull request Closes #1970 from yaooqinn/cdn. Closes #1970 a325f828 [Kent Yao] Repalce mirror-cn profile with Apache officially suggested dlcdn a0756b4f [Kent Yao] Repalce mirror-cn profile with Apache officially suggested dlcdn Authored-by:
-
ulysses-you authored
### _Why are the changes needed?_ We have `TFramedTransport` in test of `kyuubi-extension-spark-common` and the `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2` have already dependent on the test-jar of `kyuubi-extension-spark-common`. So it's redundant, and we can remove `TFramedTransport` in `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2` ### _How was this patch tested?_ Pass CI Closes #1971 from ulysses-you/KYUUBI-1948-FOLLOWUP. Closes #1948 43193606 [ulysses-you] remove TFramedTransport in extension spark-3-1 and spark-3-2 Authored-by:
ulysses-you <ulyssesyou18@gmail.com> Signed-off-by:
-
