Summary:
This check wasn't handling the fact that we might see the same update
multiple times. As a result we were requesting missing updates with equal from
and to values and holding on to that connection for 20 minutes until it timed
out. This should get rid of such spurious requests.

Test Plan:
The async nature of this and double delivery of stan is hard to
reproduce in a test environment. I think it's doable but will require some
rewriting of the existing test. Will be added in a followup.

Reviewers: michelle, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7589

GitOrigin-RevId: 56f52f4538563619458f6ac3d7b1f611f6924f5d

Summary: It's cleaner and more correct.

Test Plan: Existing tests.

Reviewers: yzhao, jps, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7584

GitOrigin-RevId: 4674266952f1fbd04523fd505567029e71cd1957

Summary:
More modular approach than the previous version.

Triggered by a potential second use of this functionality, but turns out it's not needed. Still, this is a much cleaner and more modular approach, so let's keep it.

Test Plan: Existing tests

Reviewers: yzhao, jps, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7575

GitOrigin-RevId: 29715ff07b536fd1e43263967a298b8cb4965908

Summary:
Adds udfs/udas for clustering request paths into unique endpoints. For instance, "/product/ABCD", "/product/EFGH"... will get clustered into one "/product/*" endpoint.

These should be considered beta functions because there are some performance issues associated with using them.

Separate diff on public github repo for beta scripts to use these udfs/udas.

Test Plan: Added some basic tests, and tested with scripts in github diff.

Reviewers: nserrino, zasgar, #engineering

Reviewed By: nserrino, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7559

GitOrigin-RevId: e21781d05ff16a1fd95b6a3e16db90fd459994f1

Summary: Rescan: exponential back-off per PID

Test Plan: dyn_lib_trace_bpf_test

Reviewers: #engineering, yzhao

Reviewed By: #engineering, yzhao

Subscribers: yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D7303

GitOrigin-RevId: 56ed7224e4657d0642612f13684ae85b03e68c04

Summary: Vihang left a comment on D7579 after I landed it to simply the return statement in the query result forwarder. This diff addresses that and also ensures that a message that I needed when debugging a New Relic cluster will be printed out.

Test Plan: existing

Reviewers: vihang, michelle, #engineering

Reviewed By: vihang, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7586

GitOrigin-RevId: 6ee0445b07a4e8d5562e82f144e1c9aee635846b

Summary:
There is a small risk that when CIDR blocks were never updated, such connections would stay open for
ever. But that should be extremely rare, and if that happens, MDS probably was busted anyway,
which would be a more severe issue.

Test Plan: Manual testing

Reviewers: oazizi, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7560

GitOrigin-RevId: 133089bff1035ad100984703db778c4a8a44af42

Summary:
AKS clusters work with pixie, so we should add it to the cli allowlist. checking for aks clusters is more annoying than gke/eks clusters, since they dont have a special k8s version we can just check.
instead we have to take the same route that we do with minikube, where we check the azure CLI to see if the cluster is in its list of clusters.

Test Plan: created aks cluster and made sure it allowed me to deploy without warnings

Reviewers: zasgar, oazizi, vihang, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7585

GitOrigin-RevId: 97ea84fccfd0f681e79a5d303a72121ed1010ec8

Summary: Name is too long. Shorten.

Test Plan: Existing tests.

Reviewers: yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7577

GitOrigin-RevId: c56c976f309c08449df23062ad1acf65d2a25d90

Summary: Follow-up to recent interface to BCCWrapper. We can now remove direct access to BCC.

Test Plan: Existing tests

Reviewers: yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7580

GitOrigin-RevId: c6f1eb468b7b419dd6f02ea1fee153bc864f9fc3

Test Plan: Jenkins

Reviewers: jps, oazizi, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7582

GitOrigin-RevId: d47960e309ba50806a5519558e0a30f3bdad05b2

Test Plan:
Manually tested with this step inside the main diff build, and verify that the deployment was done.

The deployment can fail because of pixie deployment not finished successfully, for example,
some pods are not healthy. We'll gradually improve the situation in the future.

Reviewers: oazizi, #engineering, vihang

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7504

GitOrigin-RevId: 7abbf2ea29def2f2d5c31b243c83da8973230743

Summary:
When debugging the metadata UDF seg fault, it became evident that even if there is an error on the agent side, the query broker will always respond with an error that says that the client (aka UI or CLI) closed the stream. This was because of the following sequence of action that resulted in the actual error being silenced:
1. Error on the agent side triggers the signal to close the client stream
2. Closing the client stream
3. Returning an error that the client stream has been closed, even though it was only closed as the result of a non-client related error.
This diff ensures that the errors are actually propagated correctly and not silenced.

Test Plan: edited tests

Reviewers: vihang, michelle, zasgar, #engineering

Reviewed By: michelle, #engineering

JIRA Issues: PP-2486

Differential Revision: https://phab.corp.pixielabs.ai/D7579

GitOrigin-RevId: a86cc746e874a694287a47259c112aac30ff4658

Summary: Fixes bugs with upgrade by changing persistent metadata deployment to a statefulset. Also separates the kustomize file order.

Test Plan: Manual. px deploy with persistent store, then upgraded with skaffold to check it was working.

Reviewers: vihang, nserrino, michelle, #engineering

Reviewed By: nserrino, michelle, #engineering

JIRA Issues: PP-2483

Differential Revision: https://phab.corp.pixielabs.ai/D7574

GitOrigin-RevId: cd1dfc681017fcb34e5966569e2c4feab73d3787

Summary: Large clusters are more likely to have lots of k8s metadata which eventually gets expired. We were noticing that large clusters had queries that would never complete. I determined that this was due to a seg fault in Kelvin (problem 1) that was then handled weirdly by the query broker (problem 2). This diff fixes problem 1, the seg faulting in metadata UDFs causing a Kelvin crash when a pod that we didn't have metadata for was queries. Problem 2 will be fixed in a subsequent diff.

Test Plan: added unit tests for missing/invalid inputs.

Reviewers: michelle, vihang, vjain, zasgar, #engineering

Reviewed By: michelle, vihang, #engineering

JIRA Issues: PP-2486

Differential Revision: https://phab.corp.pixielabs.ai/D7578

GitOrigin-RevId: 0228789fd2c44629fc64e676303994f882c574d6

Summary: GetProcExe now returns Status: Was useful in debugging an issue, so let's keep it.

Test Plan: Existing tests.

Reviewers: yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7568

GitOrigin-RevId: acf124f2e22b6d202227c440cbea6e19139e72dc

Summary: It increasingly becomes annoying to typing the full word.

Test Plan: Jenkins

Reviewers: oazizi, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7566

GitOrigin-RevId: 222d67bddebb5fa649376e4b51b527c9f02008e9

Summary: This flag was removed, but there are some old references that were not caught. Remove them.

Test Plan: Jenkins

Reviewers: nserrino, yzhao, #engineering

Reviewed By: nserrino, yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7576

GitOrigin-RevId: 5bbb81db3c550ec158cd3b3ff8222ba890d69737

Summary:
Expand BCCWrapper to include table access functions.

Goal is uniformity--unformity of access (eventually, we will be able to get rid of `BCCWrapper::bpf()`),
and unformity of Status return values.

Test Plan: Existing tests

Reviewers: yzhao, jps, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7573

GitOrigin-RevId: 4c94a5464f42861f14b72e2a787842df9852a642

Summary: Sharing code, and standardizing.

Test Plan: Existing tests.

Reviewers: yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7569

GitOrigin-RevId: b4e9b1ada92632f98f78f2c7900e869ae5342eaf

Summary:
Helps keep the code more concise.

We can use it in other instances too, but let's start with the core.

Test Plan: Existing tests.

Reviewers: yzhao, jps, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7570

GitOrigin-RevId: bfaa8139f69550adcea4a7f975dca3110445c032

Summary:
Before this bazel build //src/stirling/bpf_tools/... would try to compile all of stirling because of the probe_cleaner_test.

As such, seems better suited to live in the e2e_tests directory.

Test Plan: Jenkins

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7564

GitOrigin-RevId: e11f9dfda5f54f882bd70b260ca6c821c88e292d

Summary:
A yaml fix, and some other script & documentation fixes for consistency.

Mostly around --host=pid

Test Plan: Existing tests.

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7561

GitOrigin-RevId: 31535b88ebe639598ae5800e608e2218817b89e5

Summary: Typo. Someone with prod access needs to run the script.

Test Plan: Once prod is updated, check headers when fetching these bundles.

Reviewers: zasgar, michelle, nserrino, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7572

GitOrigin-RevId: 838dc860fc174fb1ef131822c03753b5e3123024

Summary: D7536 was meant to allow [scripts with required args](https://github.com/pixie-labs/pixie/pull/211/files) to work. It missed a spot that could cause a whitescreen.

Test Plan: Run locally. [This link](https://withpixie.ai:8080/live/clusters/gke_pl-pixies_us-west1-a_dev-jps/namespaces/pl?start_time=-5m) should no longer whitescreen.

Reviewers: #engineering, nserrino, michelle, vihang

Reviewed By: #engineering, nserrino, michelle

Differential Revision: https://phab.corp.pixielabs.ai/D7571

GitOrigin-RevId: f17b5363537fde7eed201a88eb392ffb6ff62d6d

Summary: This follows D7525.

Test Plan: This is more tests.

Reviewers: yzhao, jps, rcheng, #engineering

Reviewed By: yzhao, #engineering

JIRA Issues: PP-2448

Differential Revision: https://phab.corp.pixielabs.ai/D7549

GitOrigin-RevId: d405efc463302bafa8c80a0182c0a32ee5a2438e

Summary: This diff enables the TaskStructResolver for BCC instances that can't find Linux headers on the host.

Test Plan: Jenkins is not a good test environment, because it does have host headers. Still thinking of ways of testing this. Will not release until more rigorously tested.

Reviewers: yzhao, jps, rcheng, #engineering

Reviewed By: jps

Differential Revision: https://phab.corp.pixielabs.ai/D7525

GitOrigin-RevId: 5a8feba523770e63d089226d13225dfbab831f72

Summary: Some scripts for deploying on k8s.

Test Plan: Manual

Reviewers: yzhao, jps, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7545

GitOrigin-RevId: 30002a264c655d41878a195ad584a1f0e474ead3

Summary:
Made this to investigate the performance issues on gke:prod,
but good to have around.

Test Plan: None

Reviewers: jps, yzhao, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7537

GitOrigin-RevId: 0146c01421f82dd290409d2f2d98e9cf4761aeee

Summary: Tidying up.

Test Plan: Existing tests.

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7555

GitOrigin-RevId: beb0e17fcc1c690f2d3c893de1eddce944b96471

Summary: passthrough proxy currently returns any query GRPC error as a stream read error, which masks the actual error and makes it harder to debug problems with the query broker. non-passthrough mode will just return the error, so let's do the same thing here.

Test Plan: edited

Reviewers: michelle, vihang, #engineering

Reviewed By: vihang, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7556

GitOrigin-RevId: dbae85153735aeef7421bf0c1b87c1ac99debfad

Summary:
This prevents multiple missing request responses from relying on
the same topic/channels.

Test Plan:
skaffld deploy cloud, restart the vzconn and vzmgr pods. Make sure
everything recovers as expected

Reviewers: michelle, #engineering

Reviewed By: michelle, #engineering

Subscribers: nserrino

Differential Revision: https://phab.corp.pixielabs.ai/D7548

GitOrigin-RevId: b23f375a244fc938cbaac1b7ffc1804866c9369d

Summary: Just tidying up a bit. Forked off from another diff.

Test Plan: Existing tests.

Reviewers: yzhao, jps, rcheng, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7550

GitOrigin-RevId: ec6902327d868ecaddca38646d175131d029f75b

Summary: One node on `gke:prod` was using a lot of CPU. Turns out its related to how we deal with failed protobuf parsing. Fix it.

Test Plan: Existing tests.

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7551

GitOrigin-RevId: cbc013d481e63f0d83b1bafaca3a0c16cf80d105

Summary: Depends on D7528, which added UDF source executor as a concept to scalar UDFs, re-using the same approach as UDTF. This diff adds the ability to the registry info class to look up which UDF Source executor type a given scalar UDF has.

Test Plan: added unit test

Reviewers: philkuz, zasgar, jamesbartlett, michelle, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2445

Differential Revision: https://phab.corp.pixielabs.ai/D7530

GitOrigin-RevId: 8518fd3bbddeeaf6e32ccb5159c41ca5b2e405fc

Summary:
An argument to a script is required if its variable definition omits `defaultValue`. The UI shows this with an asterisk. This diff also refuses to run the script, and warns the user, if they try to run a script with such an argument missing.

In this image, `defaultValue` is omitted in the vis spec, the argument is marked with an asterisk, the value is missing, and the run button was pressed.
{F115650}

Test Plan: Open a script, remove the `defaultValue` field from one of its variables in the vis spec, and then try to run it with the arg missing. It should refuse, and tell you to specify the arg.

Reviewers: michelle, vihang, #engineering

Reviewed By: michelle, #engineering

Subscribers: nserrino

JIRA Issues: PC-548

Differential Revision: https://phab.corp.pixielabs.ai/D7536

GitOrigin-RevId: 0d9679bafe0c3f250a8e78af86a45d0f1f474977

Test Plan: Jenkins

Reviewers: oazizi, nserrino, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7553

GitOrigin-RevId: 976791bdc0380f799170fe46b7b8eb4211ba05d7

Summary: Similar to UDTFs, some UDFs should only be executed on Kelvin or PEM Carnot instances. This diff follows the same protocol used for UDTF in order to add that distinction for UDF. This will be used to help plan queries so that Kelvin-only UDFs are scheduled on Kelvin, and also alert the user with a compiler error if they are trying to run a PEM-only UDF after a blocking node (aka on a Kelvin node). While some aspects of how we handle metadata and compilation of distributed plans will change in the future with Carnot ETL, the concept of certain UDFs needing to execute in on a specific type of agent is expected to stick around in some form.

Test Plan: updated unit tests

Reviewers: philkuz, zasgar, jamesbartlett, michelle, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2445

Differential Revision: https://phab.corp.pixielabs.ai/D7528

GitOrigin-RevId: 2e8674fd914978926bb3f0e52831f175acb76bcb

Summary: TSIA. Also we no longer need `beta` for container commands.

Test Plan: Run chef

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7472

GitOrigin-RevId: 552f28a2357038aca81e8b777f3cb0fff9607113

Summary:
This makes it easier to grab a profile on demand if there are any
problems with the service.
Note, if running `go tool pprof` on a port forwarded service, use `https+insecure`
as the scheme to ignore the TLS cert.

Test Plan:
Deploy pixie cloud with skaffold. Portforward one of the services.
Run `go tool pprof`, ensure that we can inspect profiles.

Reviewers: zasgar, michelle, #engineering

Reviewed By: zasgar, #engineering

Subscribers: yzhao

Differential Revision: https://phab.corp.pixielabs.ai/D7538

GitOrigin-RevId: c8db56d7dcb0afe0164bebf2180c52432ed4972b