Summary:
There is a small risk that when CIDR blocks were never updated, such connections would stay open for
ever. But that should be extremely rare, and if that happens, MDS probably was busted anyway,
which would be a more severe issue.

Test Plan: Manual testing

Reviewers: oazizi, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7560

GitOrigin-RevId: 133089bff1035ad100984703db778c4a8a44af42

Summary:
AKS clusters work with pixie, so we should add it to the cli allowlist. checking for aks clusters is more annoying than gke/eks clusters, since they dont have a special k8s version we can just check.
instead we have to take the same route that we do with minikube, where we check the azure CLI to see if the cluster is in its list of clusters.

Test Plan: created aks cluster and made sure it allowed me to deploy without warnings

Reviewers: zasgar, oazizi, vihang, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7585

GitOrigin-RevId: 97ea84fccfd0f681e79a5d303a72121ed1010ec8

Summary: Name is too long. Shorten.

Test Plan: Existing tests.

Reviewers: yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7577

GitOrigin-RevId: c56c976f309c08449df23062ad1acf65d2a25d90

Summary: Follow-up to recent interface to BCCWrapper. We can now remove direct access to BCC.

Test Plan: Existing tests

Reviewers: yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7580

GitOrigin-RevId: c6f1eb468b7b419dd6f02ea1fee153bc864f9fc3

Test Plan: Jenkins

Reviewers: jps, oazizi, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7582

GitOrigin-RevId: d47960e309ba50806a5519558e0a30f3bdad05b2

Test Plan:
Manually tested with this step inside the main diff build, and verify that the deployment was done.

The deployment can fail because of pixie deployment not finished successfully, for example,
some pods are not healthy. We'll gradually improve the situation in the future.

Reviewers: oazizi, #engineering, vihang

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7504

GitOrigin-RevId: 7abbf2ea29def2f2d5c31b243c83da8973230743

Summary:
When debugging the metadata UDF seg fault, it became evident that even if there is an error on the agent side, the query broker will always respond with an error that says that the client (aka UI or CLI) closed the stream. This was because of the following sequence of action that resulted in the actual error being silenced:
1. Error on the agent side triggers the signal to close the client stream
2. Closing the client stream
3. Returning an error that the client stream has been closed, even though it was only closed as the result of a non-client related error.
This diff ensures that the errors are actually propagated correctly and not silenced.

Test Plan: edited tests

Reviewers: vihang, michelle, zasgar, #engineering

Reviewed By: michelle, #engineering

JIRA Issues: PP-2486

Differential Revision: https://phab.corp.pixielabs.ai/D7579

GitOrigin-RevId: a86cc746e874a694287a47259c112aac30ff4658

Summary: Fixes bugs with upgrade by changing persistent metadata deployment to a statefulset. Also separates the kustomize file order.

Test Plan: Manual. px deploy with persistent store, then upgraded with skaffold to check it was working.

Reviewers: vihang, nserrino, michelle, #engineering

Reviewed By: nserrino, michelle, #engineering

JIRA Issues: PP-2483

Differential Revision: https://phab.corp.pixielabs.ai/D7574

GitOrigin-RevId: cd1dfc681017fcb34e5966569e2c4feab73d3787

Summary: Large clusters are more likely to have lots of k8s metadata which eventually gets expired. We were noticing that large clusters had queries that would never complete. I determined that this was due to a seg fault in Kelvin (problem 1) that was then handled weirdly by the query broker (problem 2). This diff fixes problem 1, the seg faulting in metadata UDFs causing a Kelvin crash when a pod that we didn't have metadata for was queries. Problem 2 will be fixed in a subsequent diff.

Test Plan: added unit tests for missing/invalid inputs.

Reviewers: michelle, vihang, vjain, zasgar, #engineering

Reviewed By: michelle, vihang, #engineering

JIRA Issues: PP-2486

Differential Revision: https://phab.corp.pixielabs.ai/D7578

GitOrigin-RevId: 0228789fd2c44629fc64e676303994f882c574d6

Summary: GetProcExe now returns Status: Was useful in debugging an issue, so let's keep it.

Test Plan: Existing tests.

Reviewers: yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7568

GitOrigin-RevId: acf124f2e22b6d202227c440cbea6e19139e72dc

Summary: It increasingly becomes annoying to typing the full word.

Test Plan: Jenkins

Reviewers: oazizi, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7566

GitOrigin-RevId: 222d67bddebb5fa649376e4b51b527c9f02008e9

Summary: This flag was removed, but there are some old references that were not caught. Remove them.

Test Plan: Jenkins

Reviewers: nserrino, yzhao, #engineering

Reviewed By: nserrino, yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7576

GitOrigin-RevId: 5bbb81db3c550ec158cd3b3ff8222ba890d69737

Summary:
Expand BCCWrapper to include table access functions.

Goal is uniformity--unformity of access (eventually, we will be able to get rid of `BCCWrapper::bpf()`),
and unformity of Status return values.

Test Plan: Existing tests

Reviewers: yzhao, jps, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7573

GitOrigin-RevId: 4c94a5464f42861f14b72e2a787842df9852a642

Summary: Sharing code, and standardizing.

Test Plan: Existing tests.

Reviewers: yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7569

GitOrigin-RevId: b4e9b1ada92632f98f78f2c7900e869ae5342eaf

Summary:
Helps keep the code more concise.

We can use it in other instances too, but let's start with the core.

Test Plan: Existing tests.

Reviewers: yzhao, jps, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7570

GitOrigin-RevId: bfaa8139f69550adcea4a7f975dca3110445c032

Summary:
Before this bazel build //src/stirling/bpf_tools/... would try to compile all of stirling because of the probe_cleaner_test.

As such, seems better suited to live in the e2e_tests directory.

Test Plan: Jenkins

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7564

GitOrigin-RevId: e11f9dfda5f54f882bd70b260ca6c821c88e292d

Summary:
A yaml fix, and some other script & documentation fixes for consistency.

Mostly around --host=pid

Test Plan: Existing tests.

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7561

GitOrigin-RevId: 31535b88ebe639598ae5800e608e2218817b89e5

Summary: Typo. Someone with prod access needs to run the script.

Test Plan: Once prod is updated, check headers when fetching these bundles.

Reviewers: zasgar, michelle, nserrino, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7572

GitOrigin-RevId: 838dc860fc174fb1ef131822c03753b5e3123024

Summary: D7536 was meant to allow [scripts with required args](https://github.com/pixie-labs/pixie/pull/211/files) to work. It missed a spot that could cause a whitescreen.

Test Plan: Run locally. [This link](https://withpixie.ai:8080/live/clusters/gke_pl-pixies_us-west1-a_dev-jps/namespaces/pl?start_time=-5m) should no longer whitescreen.

Reviewers: #engineering, nserrino, michelle, vihang

Reviewed By: #engineering, nserrino, michelle

Differential Revision: https://phab.corp.pixielabs.ai/D7571

GitOrigin-RevId: f17b5363537fde7eed201a88eb392ffb6ff62d6d

Summary: This follows D7525.

Test Plan: This is more tests.

Reviewers: yzhao, jps, rcheng, #engineering

Reviewed By: yzhao, #engineering

JIRA Issues: PP-2448

Differential Revision: https://phab.corp.pixielabs.ai/D7549

GitOrigin-RevId: d405efc463302bafa8c80a0182c0a32ee5a2438e

Summary: This diff enables the TaskStructResolver for BCC instances that can't find Linux headers on the host.

Test Plan: Jenkins is not a good test environment, because it does have host headers. Still thinking of ways of testing this. Will not release until more rigorously tested.

Reviewers: yzhao, jps, rcheng, #engineering

Reviewed By: jps

Differential Revision: https://phab.corp.pixielabs.ai/D7525

GitOrigin-RevId: 5a8feba523770e63d089226d13225dfbab831f72

Summary: Some scripts for deploying on k8s.

Test Plan: Manual

Reviewers: yzhao, jps, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7545

GitOrigin-RevId: 30002a264c655d41878a195ad584a1f0e474ead3

Summary:
Made this to investigate the performance issues on gke:prod,
but good to have around.

Test Plan: None

Reviewers: jps, yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7537

GitOrigin-RevId: 0146c01421f82dd290409d2f2d98e9cf4761aeee

Summary: Tidying up.

Test Plan: Existing tests.

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7555

GitOrigin-RevId: beb0e17fcc1c690f2d3c893de1eddce944b96471

Summary: passthrough proxy currently returns any query GRPC error as a stream read error, which masks the actual error and makes it harder to debug problems with the query broker. non-passthrough mode will just return the error, so let's do the same thing here.

Test Plan: edited

Reviewers: michelle, vihang, #engineering

Reviewed By: vihang, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7556

GitOrigin-RevId: dbae85153735aeef7421bf0c1b87c1ac99debfad

Summary:
This prevents multiple missing request responses from relying on
the same topic/channels.

Test Plan:
skaffld deploy cloud, restart the vzconn and vzmgr pods. Make sure
everything recovers as expected

Reviewers: michelle, #engineering

Reviewed By: michelle, #engineering

Subscribers: nserrino

Differential Revision: https://phab.corp.pixielabs.ai/D7548

GitOrigin-RevId: b23f375a244fc938cbaac1b7ffc1804866c9369d

Summary: Just tidying up a bit. Forked off from another diff.

Test Plan: Existing tests.

Reviewers: yzhao, jps, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7550

GitOrigin-RevId: ec6902327d868ecaddca38646d175131d029f75b

Summary: One node on `gke:prod` was using a lot of CPU. Turns out its related to how we deal with failed protobuf parsing. Fix it.

Test Plan: Existing tests.

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7551

GitOrigin-RevId: cbc013d481e63f0d83b1bafaca3a0c16cf80d105

Summary: Depends on D7528, which added UDF source executor as a concept to scalar UDFs, re-using the same approach as UDTF. This diff adds the ability to the registry info class to look up which UDF Source executor type a given scalar UDF has.

Test Plan: added unit test

Reviewers: philkuz, zasgar, jamesbartlett, michelle, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2445

Differential Revision: https://phab.corp.pixielabs.ai/D7530

GitOrigin-RevId: 8518fd3bbddeeaf6e32ccb5159c41ca5b2e405fc

Summary:
An argument to a script is required if its variable definition omits `defaultValue`. The UI shows this with an asterisk. This diff also refuses to run the script, and warns the user, if they try to run a script with such an argument missing.

In this image, `defaultValue` is omitted in the vis spec, the argument is marked with an asterisk, the value is missing, and the run button was pressed.
{F115650}

Test Plan: Open a script, remove the `defaultValue` field from one of its variables in the vis spec, and then try to run it with the arg missing. It should refuse, and tell you to specify the arg.

Reviewers: michelle, vihang, #engineering

Reviewed By: michelle, #engineering

Subscribers: nserrino

JIRA Issues: PC-548

Differential Revision: https://phab.corp.pixielabs.ai/D7536

GitOrigin-RevId: 0d9679bafe0c3f250a8e78af86a45d0f1f474977

Test Plan: Jenkins

Reviewers: oazizi, nserrino, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7553

GitOrigin-RevId: 976791bdc0380f799170fe46b7b8eb4211ba05d7

Summary: Similar to UDTFs, some UDFs should only be executed on Kelvin or PEM Carnot instances. This diff follows the same protocol used for UDTF in order to add that distinction for UDF. This will be used to help plan queries so that Kelvin-only UDFs are scheduled on Kelvin, and also alert the user with a compiler error if they are trying to run a PEM-only UDF after a blocking node (aka on a Kelvin node). While some aspects of how we handle metadata and compilation of distributed plans will change in the future with Carnot ETL, the concept of certain UDFs needing to execute in on a specific type of agent is expected to stick around in some form.

Test Plan: updated unit tests

Reviewers: philkuz, zasgar, jamesbartlett, michelle, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2445

Differential Revision: https://phab.corp.pixielabs.ai/D7528

GitOrigin-RevId: 2e8674fd914978926bb3f0e52831f175acb76bcb

Summary: TSIA. Also we no longer need `beta` for container commands.

Test Plan: Run chef

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7472

GitOrigin-RevId: 552f28a2357038aca81e8b777f3cb0fff9607113

Summary:
This makes it easier to grab a profile on demand if there are any
problems with the service.
Note, if running `go tool pprof` on a port forwarded service, use `https+insecure`
as the scheme to ignore the TLS cert.

Test Plan:
Deploy pixie cloud with skaffold. Portforward one of the services.
Run `go tool pprof`, ensure that we can inspect profiles.

Reviewers: zasgar, michelle, #engineering

Reviewed By: zasgar, #engineering

Subscribers: yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D7538

GitOrigin-RevId: c8db56d7dcb0afe0164bebf2180c52432ed4972b

Summary:
Previously, semantic type rules involved manually specifying the semantic types that should be inferred from a UDF's arguments. This can be a bit tedious for UDFs like px.select and px.any, which will always propagate their input arguments (as long as they match, in px.select's case). This diff extends the InheritTypeFromArgs rule in order to generate a list of all the current SemanticTypes, so that UDFs like px.select don't have to enumerate every existing UDF every time.
This is to support htroisi's work improving the redis_flow_graph live view.

Test Plan: tested it on cluster

Reviewers: jamesbartlett, zasgar, philkuz, #engineering

Reviewed By: jamesbartlett, #engineering

Subscribers: htroisi

Differential Revision: https://phab.corp.pixielabs.ai/D7513

GitOrigin-RevId: c5ce2905ee823f5117bfb30602643cb976deb5d8

Summary: While addressing PP-2445, I noticed that there was unused distributed splitter code, so this diff removes that code.

Test Plan: existing

Reviewers: philkuz, #engineering, zasgar

Reviewed By: #engineering, zasgar

Differential Revision: https://phab.corp.pixielabs.ai/D7544

GitOrigin-RevId: 84596f3295280df3642fcc61d6bf22254eeaa6cb

Summary: TSIA

Test Plan: N/A

Reviewers: zasgar, michelle, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7546

GitOrigin-RevId: 3170c64178bc0951fb7083a2cd2f5b41b9082e71

Summary:
this is for the profiler project. the input columns from the profiler are stacktrace and count.
the frontend will be responsible for processing these columns and creating a JSON which can be used by Vega.
this diff contains only the vispec so far.

Test Plan: n/a

Reviewers: nserrino, nlanam, jamesbartlett, philkuz, #engineering

Reviewed By: nserrino, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7540

GitOrigin-RevId: 3dcbdeabdc9a5d43f6ccf1fce01bebaf559df460

Summary: TSIA

Test Plan: Run `arc lint` from within `src/ui`

Reviewers: nlanam, #engineering

Reviewed By: nlanam, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7547

GitOrigin-RevId: 69e146238edaddd9c6475227dfa87a341b3edd63

Summary:
Previously, we were running the `eslint_ui.sh` helper script as part of the version check even if we didn't have any files to lint with `eslint`.
To speed it up, I removed the version check.

However arcanist runs linters in parallel, which now meant that we had multiple `yarn install`s running concurrently.
These would stomp on the contents of `node_modules` and the `eslint` binary causing some lint runs to emit no output which in turn cause `arc` to fatal.

Instead we hook into `willLintPaths` and run a setup script once if there are any paths before calling the parent class's `willLintPaths` which will actually create the futures to run the linter.

Test Plan:
Before this change
- edit multiple `ts` files
- `rm -rf src/ui/node_modules; arc lint --trace`
- notice failure

After this change
- edit multiple `ts` files
- `rm -rf src/ui/node_modules; arc lint --trace`
- notice `eslint_setup.sh` being run first
- notice `eslint_ui.sh` being run in parallel after `eslint_setup.sh` exits
- no failures

Reviewers: nlanam, #engineering

Reviewed By: nlanam, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7541

GitOrigin-RevId: d92ba7a602974dbf8736df1c2191984b0fb41804