Summary:
if a vizier/cli version is specified in the artifact tracker deployment, it assumes that the version exists without having to check the db.
uesrs who are deploying a fresh pixie cloud will not need to run load_dev_db.

Test Plan: deployed to a version to staging and tested CLI updates/vizier deploys

Reviewers: vihang, zasgar

Reviewed By: vihang

Differential Revision: https://phab.corp.pixielabs.ai/D8505

GitOrigin-RevId: 983439503e3b87842c4d5a2c17552a6c92e26d97

Summary:
Install certs applies certs to the current kube config. This is a bit problematic since it clobbers the existing certs and causes running services to pick up the new certs.
Instead change into a create certs command that generates the yamls but doesn't apply them.

Also did a bunch of heavy rewriting the clean up the cert generation and split up the vizier vs cloud cluster certs.

Test Plan:
Upgraded a dev cloud. Upgraded a dev vizier.
Ran a dev cloud from scratch. Ran a dev vizier from scratch.

Reviewers: michelle, philkuz

Reviewed By: michelle

JIRA Issues: PC-875

Differential Revision: https://phab.corp.pixielabs.ai/D8488

GitOrigin-RevId: 0ff53f64ebf6f8c67b8dfae5d64ea405dafc94bb

Summary: These are no longer used

Test Plan: N/A

Reviewers: zasgar, michelle

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8498

GitOrigin-RevId: 988b66bc985c9f21de3cc73fd081131343f706a4

Set path precedence to prefer chef installed deps before falling back to /usr/local/bin and /usr/bin

Summary:
TSIA. This prevents issues caused by folks running `apt install nodejs`
and breaking the build environment for everyone.

Test Plan: N/A

Reviewers: zasgar, michelle

Reviewed By: zasgar, michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8491

GitOrigin-RevId: 5823487ac415766016c1781071542e3e65ee77dc

Summary: Disable support emails access based on flag.

Test Plan: Manual

Reviewers: michelle, nserrino

Reviewed By: michelle

JIRA Issues: PP-2670

Differential Revision: https://phab.corp.pixielabs.ai/D8497

GitOrigin-RevId: 5ab94e3a15982ca326cfec8f5e9c39eaf5bcf9f4

Summary:
we currently require users to enter cloud-dns-secrets to add DNS records for users using passthrough.
this diff updates the dnsmgr deployment so that supplying these secrets is optional. in this case, direct mode won't work properly. however, the assumption is that if they are hosting their own cloud, this is less of an issue.

there is still some inefficiency with how vzmgr requests the DNSAddress from dnsmgr. once the vizier's proxy server is ready with an IP address, it will make a request to dnsmgr every heartbeat.
i thought through a couple of ways to clean this up, for example, only making this call only when the user has direct mode enabled. but we probably need something a little more complex to satisfy these conditions:
1. DNS records take some time to propagate. if we only make the request once the user enables direct mode, there will be some period of time where they can't query their cluster.
2. we need to handle cases where the user's proxy IP address has changed, which is why we continue to request a DNS address even after the vizier already has an addressed assigned. it is currently dnsmgr's responsibility to handle changing IP addresses
given this, we probably need to think more carefully about the fix for this. for the purposes of this issue, we should just make the secrets optional.

Test Plan: test with staging

Reviewers: vihang, zasgar

Reviewed By: zasgar

Differential Revision: https://phab.corp.pixielabs.ai/D8492

GitOrigin-RevId: cf87c5349f0e2cd736c1c8ae0cb286530ec1fd1e

Summary: the test is still flaky and causing builds to fail. disabling for now until we fix it

Test Plan: n/a

Reviewers: philkuz, michelle

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8494

GitOrigin-RevId: 1cab41e404a6fa44d59c69138855901fe2218328

Summary: TSIA

Test Plan: N/A

Reviewers: jamesbartlett

Reviewed By: jamesbartlett

Differential Revision: https://phab.corp.pixielabs.ai/D8495

GitOrigin-RevId: cf94fc35ba8c3d0eb9b6acf9ba769393c2c77cbd

Summary:
- Adds px/sql_query and px/sql_queries.
- px/sql_queries shows LET stats for normalized sql queries grouped by normalized query.
- px/sql_query shows LET stats for an individual normalized query grouped by the parameter sets to that query.

Test Plan: Tested with skaffold

Reviewers: nserrino, zasgar, htroisi

Reviewed By: nserrino

JIRA Issues: PP-2658

Differential Revision: https://phab.corp.pixielabs.ai/D8482

GitOrigin-RevId: 8af5bb0badd047f3fd2dd5b4cb35abc611154b37

Summary:
- Adds code for handling the parse tree output of the ANTLR parsers.
- Adds normalize_pgsql and normalize_mysql UDFs to carnot.
- Adds px/sql_queries and px/sql_query scripts. px/sql_queries displays LET statistics grouped by normalized query. px/sql_query shows LET statistics for a specific normalized query grouped by its parameter sets.
- Adds benchmarks for the normalization functions. There's a fast path where only SLL parsing mode is required, and there's a slow path where LL* is required. I've tried to modify the grammar such that there are no cases where LL* is required, but without more extensive testing I can't say for sure that it'll always only use SLL. If only SLL is required, it takes roughly 50us. If LL* is required all bets are off and it can take 1-3ms. I have a test case showing that most basic queries will only require SLL.

Test Plan: Tested scripts with skaffold.

Reviewers: nserrino, zasgar

Reviewed By: nserrino

JIRA Issues: PP-2658, PP-2675

Differential Revision: https://phab.corp.pixielabs.ai/D8439

GitOrigin-RevId: 72054627311d309a7cfcf2724efecbd3e0b6ab41

[PxL Breaking Change][SQL Normalization] Change mysql `req` column to output structured data for EXECUTE

Summary:
Change the req column of the mysql table to output `query=[...] params=[...]` structured data for EXECUTE requests.
Before this change, the "sql" output in the req column was invalid for com_stmt_execute requests, since the parameters were inserted without regard for quoting etc.

Test Plan: Updated tests, tests pass.

Reviewers: oazizi, yzhao

Reviewed By: oazizi

JIRA Issues: PP-2659

Differential Revision: https://phab.corp.pixielabs.ai/D8472

GitOrigin-RevId: 178f2b162f62996373384082b603930dd62a2eeb

Summary:
In order to be more consistent with MySQL, and make parsing in PxL easier. The Postgres tracing now outputs an extra column with the command code, instead of inlining the command into the request body.

Eg.
Before: {'req_body': 'EXECUTE [query=[...], params=[...]]'}
After: {'req_body': 'query=[...] params=[...]', 'req_cmd': <command code for EXECUTE command>}

Test Plan: Updated stirling tests for the new format.

Reviewers: oazizi, yzhao

Reviewed By: yzhao

Subscribers: htroisi

JIRA Issues: PP-2676

Differential Revision: https://phab.corp.pixielabs.ai/D8406

GitOrigin-RevId: 510de40b901b44760a4b5cf00aa4e1783ad91b72

Summary: There is nothing shared between network_stats and process_stats, so split the code up.

Test Plan: Existing tests

Reviewers: #stirling, yzhao

Reviewed By: #stirling, yzhao

Subscribers: yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D8489

GitOrigin-RevId: 75bd925416628a262ad4be83e42b0acb21d752f5

Summary: Greeter2 is unused.

Test Plan: Jenkins

Reviewers: #stirling, yzhao

Reviewed By: #stirling, yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D8474

GitOrigin-RevId: 424ec060cdbe34d16ffd0d27b86faa2c606c3c1c

Summary: Size of PB messages was being mis-reported.

Test Plan: None

Reviewers: #stirling, yzhao

Reviewed By: #stirling, yzhao

JIRA Issues: PP-2694

Differential Revision: https://phab.corp.pixielabs.ai/D8470

GitOrigin-RevId: 9b7816c2a347a23363501e14ab8ca5fc62ff67af

Summary:
Updated segment secrets (except staging and prod @michelle) to be configmaps in configs/ directory.

The diff for the secret -> configmap
```
2c2
< kind: Secret
---
> kind: ConfigMap
4c4
<     name: segment-secrets
---
>     name: segment-config
6,7c6
< type: Opaque
< stringData:
---
> data:
```

Test Plan: deploy cloud

Reviewers: michelle, vihang

Reviewed By: vihang

Subscribers: michelle

JIRA Issues: PC-877

Differential Revision: https://phab.corp.pixielabs.ai/D8481

GitOrigin-RevId: 5a7e5a184b21eee80dded3bac4eb0ae6ab0300c1

Summary: set -e with ensure_namespace would exit unexpectedly if ns was missing. kubectl get ns returns an error code. Now we capture it and return foo in case of exit code. Migth need to clean up, dependinag on reviewers opinion.

Test Plan: tested to make sure that if ns exists, it doesn't try to create one and if it doesn't, under set -e we don't exit the containing script. The script i used to test was the cloud_prereqs one.

Reviewers: vihang, michelle

Reviewed By: vihang

Differential Revision: https://phab.corp.pixielabs.ai/D8483

GitOrigin-RevId: 8c8eb66f4b3ce1ea2ecfe454164a1823b7d25e05

Summary: Kratostest was getting flakey. Removing some of the flake points should help out. Remove the migration container call, remove the volume it needed, and just pull the kratos image instead of trying to build one on top.

Test Plan: Tested kratostest 1000 times.

Reviewers: vihang, zasgar

Reviewed By: vihang

JIRA Issues: PP-2668

Differential Revision: https://phab.corp.pixielabs.ai/D8388

GitOrigin-RevId: 2c98bddb4058f6ad49cbff3731545d5c5a4a247c

Summary: we keep getting this issue with ./script/load_dev_db.sh where a running postgres instance gets in the way. To avoid this issue, here's a solution where we get a port that's two above a typical postgres and choose that instead. Slightly better than nothing, but might benefit from a future where we error out saying the port is busy. Kinda hard to do in bash, would recommend we rewrite in golang to really do this properly.

Test Plan: Tested out and after port change, now successfully writes to k8s postgres.

Reviewers: vihang, nserrino, michelle

Reviewed By: vihang

Differential Revision: https://phab.corp.pixielabs.ai/D8487

GitOrigin-RevId: aa770ba0a9fa7c148e0a5729dcc0b4ea9cddfd86

Summary:
users should be able to deploy Pixie cloud without requiring our GCS service account credentials.
part of this work required us to make official releases public (see D8484). these releases should be fetchable from GCS without having any credentials.
rcs are still uploaded to our private bucket and will still need credentials to access.

this diff updates the artifact tracker server so that it loads the credentials only if specified. it also changes which bucket it tries to access based on whether or not the desired artifact is an official release

Test Plan:
tested on staging
1. Deleted the service account secret, verified I can still deploy Vizier with official release, update CLI with official release version
2. Redeployed service account secret, verified I can still fetch Vizier/CLI official releases. also verified deploying a vizier rc works

Reviewers: vihang, zasgar

Reviewed By: vihang

JIRA Issues: PC-872

Differential Revision: https://phab.corp.pixielabs.ai/D8485

GitOrigin-RevId: 7316f54727c21c06db3d6b0115a91631f42fb3ea

Summary:
users should be able to deploy Pixie Cloud without needing access to our GCS service account.
to do so, we need to make our official releases public. this diff updates our release build so that official releases are written to our pixie-oss bucket. rcs are still written to our private pixie-prod bucket.

Test Plan:
built some rcs with the buckets switched, to ensure that jenkins had the proper permissions for writing to GCS.
verified that the files were written to the correct bucket. (will delete them now)

Reviewers: vihang, zasgar

Reviewed By: vihang

JIRA Issues: PC-872

Differential Revision: https://phab.corp.pixielabs.ai/D8484

GitOrigin-RevId: b9cbd6dcd26d91d331eccc5ec179f5ae7238c231

Summary:
Currently the vizier-image-secrets are used in two places:
1. In cli deploys, when deploying vizier versions older than 0.5.22
2. In our template generator, which is run during releases.

the vizier-image-secrets are used in the API server's `GetImageCredentials` rpc call which only the CLI deploy calls.
given that we are now on vizier 0.7.2, we can safely clean up the CLI's handling for viziers older than 0.5.22. since this is the only consumer, we can likely also remove the `GetImageCredentials` call itself. for now I've just made the vizier-image-secrets optional, and that call will error if they dont exist.

Test Plan: ran `px deploy`

Reviewers: vihang, zasgar

Reviewed By: vihang

Differential Revision: https://phab.corp.pixielabs.ai/D8486

GitOrigin-RevId: a2144f54d08a444334b54bcabda65d16365a8a3d

Summary: Deflakes the test. Also changes the behavior in that UPIDs that are not in the context will not be pushed to the tables.

Test Plan: Existing tests.

Reviewers: #stirling, yzhao

Reviewed By: #stirling, yzhao

Subscribers: yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D8465

GitOrigin-RevId: 2634af9eb273e29c94502a6ef4afebd73ed4fd07

Summary: this should point to our OSS slackin. we also need to update the docs links once we have our OSS docs ready, since users probably wont be hosting the docs on their own domain.

Test Plan: n/a

Reviewers: vihang, zasgar

Reviewed By: vihang

Differential Revision: https://phab.corp.pixielabs.ai/D8478

GitOrigin-RevId: 7405ea810fe5c1799c4a99dada588564355ada8f

Summary: Small clean-up to the structure, such that we don't have to rely on conn_event_t and close_event_t having the same layout.

Test Plan: Existing tests

Reviewers: #stirling, yzhao

Reviewed By: #stirling, yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D8456

GitOrigin-RevId: cbea73fe9fdccc22e2ef077094f80e8918c5166f

PC-878: Moving oss_auth to ory_auth and clarify docs to show that skaffold can run multiple profiles non-exclusively

Summary: skaffold lets you run multiple profiles so adjusting that. And renaming the oss_auth stuff to ory_auth

Test Plan: deployed dev and ory_auth and it worked as intended.

Reviewers: vihang, michelle

Reviewed By: vihang

JIRA Issues: PC-878

Differential Revision: https://phab.corp.pixielabs.ai/D8477

GitOrigin-RevId: ea737d102c9433f681df04c1d3b5a3aeb2483011

Summary: This reverts commit 6016cb5af425c12dd03635d39abc9bf3b4a19032. Styling was only useful with the rest of the changes.

Test Plan: Tested on yarn dev.

Reviewers: michelle

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8479

GitOrigin-RevId: 791605b25aba343f2762d9bfbe85de2882ccbbbf

Summary: revert

Test Plan: Test on yarn dev, no problems.

Reviewers: michelle, vihang

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8473

GitOrigin-RevId: e30dd8101d65289d8c2955b9d51a9ad2cc3756ae

Summary: These scripts should fail on encountering an error.

Test Plan: N/A

Reviewers: philkuz, zasgar, michelle

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8475

GitOrigin-RevId: d22eeb178adcd63cc8a4f3f8326741ab803dcba2

Summary: this broke when i made my namespace changes. the kelvin's mds_addr flag is now only a part of the actual mds address. this should have been updated to use the fully constructed version above.

Test Plan: ran skaffold

Reviewers: vihang

Reviewed By: vihang

Differential Revision: https://phab.corp.pixielabs.ai/D8471

GitOrigin-RevId: f240bb9fc5df6323354cffbfa5053275571b84ca

Summary: sign up page gutter was leaking off the div. This fixes it.

Test Plan: yarn dev, style only change

Reviewers: nlanam, michelle

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8467

GitOrigin-RevId: 6016cb5af425c12dd03635d39abc9bf3b4a19032

Summary:
TSIA, we overuse aliases especially for the leftover code from
when our proto naming was not consistent.

Now, we can stop aliasing as many things.

Test Plan: Existing build and tests.

Reviewers: zasgar, michelle

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8466

GitOrigin-RevId: 67e26155006bde8146866b08f49ab7e464135259

Summary: TSIA

Test Plan: Ran `make help`

Reviewers: zasgar, michelle

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8463

GitOrigin-RevId: b8b8dc5c409122b324e61511056bfb3e2a3cfea5

Summary: This should work again.

Test Plan: Image builds.

Reviewers: zasgar, michelle

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8462

GitOrigin-RevId: 27c59fd4a6edb879444e6753b2305e3f2c80b4b2

Summary: Remove those trackers ReadyForDestruction from the list of trackers, but wait for enough expired trackers to pay the cost of iteratiing and modifying the map.

Test Plan: Existing tests

Reviewers: #stirling, yzhao

Reviewed By: #stirling, yzhao

Subscribers: jps, yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D8411

GitOrigin-RevId: da8e9196ccf4bcc6780eed0abca91d7e7f152d38

Summary: Needs a space.

Test Plan: Manual

Reviewers: #stirling, yzhao

Reviewed By: #stirling, yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D8458

GitOrigin-RevId: 75c2dd5335898238e14697c75dde8de652cc7dfb

Summary:
--sources is repurposed to accept source names
--source_group is added to work as the old --sources flag.

Test Plan: Manual run stirling_wrapper

Reviewers: #stirling, oazizi

Reviewed By: #stirling, oazizi

Subscribers: oazizi

Differential Revision: https://phab.corp.pixielabs.ai/D8438

GitOrigin-RevId: 64308654966efe780996d9d10a76b46a6ee5d18a

Summary:
This helps quickly identify the logical connection from k8s entities
to records.

Test Plan: Manually run

Reviewers: #stirling, #engineering, philkuz

Reviewed By: #engineering, philkuz

Differential Revision: https://phab.corp.pixielabs.ai/D8461

GitOrigin-RevId: f9c8e2d02a555f63f720cfc792d9b0b1d8c18bd1

Summary:
This diff sets up user/password for Auth0. Created a new button, use the [[ https://auth0.com/docs/universal-login | Universal Login flow ]], then also needs a signup-complete page that will be redirected from a user's verification link.

Note that we have to add special set up for the auth0 account to make sure this flow is secure. Those are documented in this notion doc: https://www.notion.so/pixielabs/Creating-an-auth0-account-and-setting-up-Pixie-47b6fa663da1495cbad474139c747bea

Some design decisions: Because we block sign-up until verificaton, we cover org solution for now. Eventually we will need better support for creating orgs, likely something that will look more like Slack's model. Because of this design issue, we do not yet have support for inviting users who have already signed up for their own org.

Test Plan: Tested on dev and prod, was able to create a user, get an email verification scheme, then login.

Reviewers: michelle, vjain, nlanam

Reviewed By: michelle

JIRA Issues: PC-863

Differential Revision: https://phab.corp.pixielabs.ai/D8452

GitOrigin-RevId: 1376f2939a6e5bcb9333b34130dc199fac0592e5

Summary: load_dev_db.sh was failing. It turns out we try to end the kubectl port-forward process twice, once with a TERM signal and once with a KILL signal. In the event that the TERM signal works, the kill command will fail. This diff makes it so that we only escalate to a kill if the TERM didn't work.

Test Plan: ran the script, it now succeeds

Reviewers: vihang, michelle, philkuz

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D8459

GitOrigin-RevId: 7106408c5d71ca20c801b1e62a2f1351e7852ec0