• ulysses-you's avatar
    [KYUUBI #631] Add kyuubi-spark-extensions module · 43f40dcb
    ulysses-you authored
    <!--
    Thanks for sending a pull request!
    
    Here are some tips for you:
      1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
      2. If the PR is related to an issue in https://github.com/NetEase/kyuubi/issues
    
    , add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
      3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
    -->
    
    ### _Why are the changes needed?_
    <!--
    Please clarify why the changes are needed. For instance,
      1. If you add a feature, you can talk about the use case of it.
      2. If you fix a bug, you can clarify why it is a bug.
    -->
    The added sql module structure looks like:
    ```
    kyuubi
      |
      - dev
          |
          - kyuubi-extension-spark_3.1
    ```
    
    This PR mainly add 3 feature:
    * merging small files automatically (include dynamic partition insertion case)
    * insert shuffle node before Join to make AQE `OptimizeSkewedJoin` work
    * stage level config isolation in AQE
    
    Note that,  the sql rule  depend on the Apache Spark interface so we need make the sql module verion independence. Currently, this PR only supports the Spark 3.1.1.
    
    Due to the version issue, we need to check and deploy this extension manually currently.
    
    ### _How was this patch tested?_
    Add new test.
    
    Closes #631 from ulysses-you/add-sql-module.
    
    Closes #631
    
    2cf12f1 [ulysses-you] version
    cfbf72c [ulysses-you] address comment
    7740ca6 [ulysses-you] module name
    0f723eb [ulysses-you] workflow
    45c23d8 [ulysses-you] line
    80378f5 [ulysses-you] assembly
    95528aa [ulysses-you] move module
    5fe5d87 [ulysses-you] license
    6578440 [ulysses-you] init work
    Authored-by: default avatarulysses-you <ulyssesyou18@gmail.com>
    Signed-off-by: default avatarKent Yao <yao@apache.org>
    43f40dcb