Summary:
we saw a case in customer where deleting objects took longer than the 5 min timeout, which caused the updater job to just exit.
instead, if theres a timeout error, we should try to proceed as normal. it should be OK to apply a YAML when a previous one is still deleting (unless deleting a namespace, which we aren't in this case).
also made the timeout a little longer, which should help. however, k8s pods can be known to take hours to terminate sometimes.

Test Plan: n/a

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6099

GitOrigin-RevId: 20bb25ef8c583bb1b3c0a9193cb23f62317e0853

Summary: tsia

Test Plan: ran CLI

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6090

GitOrigin-RevId: 4e25aba747b1f15e286addb9ae30e4774365085b

Summary: Depends on D6089. Add the rule to convert StreamIR nodes into setting streaming=true on their parent memory sources to the analyzer. It should run before the rule that automatically adds a limit (which it does), since that rule will skip over DataFrames that have streaming MemorySource ancestors.

Test Plan: unit

Reviewers: philkuz, jamesbartlett, zasgar, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2161

Differential Revision: https://phab.corp.pixielabs.ai/D6095

GitOrigin-RevId: 407d1922d53b587e9cdfd15e19051896e359aea8

Summary: Streaming queries that are map only shouldn't be subject to a limit because they can execute indefinitely. In the future we may want to apply a limit per window for streaming windowed queries, which are not currently supported.

Test Plan: added unit test

Reviewers: philkuz, jamesbartlett, zasgar, #engineering

Reviewed By: zasgar, #engineering

JIRA Issues: PP-2161

Differential Revision: https://phab.corp.pixielabs.ai/D6089

GitOrigin-RevId: e3a14bdbbd322ef2a2b000a237fcd1d2c557ea83

Summary: Compile time docs should be included as part of the rendering pipeline. Add them with this.

Test Plan: Tested and outputs the copmile time functions into output.json

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6100

GitOrigin-RevId: c832801e329396c1c68b8fe7ea1dc769097a00b9

Summary: Although we plan on manually editing the DataFrame docs, it helps to write it in one place so we can save ourselves the effort before we write them out.

Test Plan: Tested and can extract them through the docstring integration thing

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6097

GitOrigin-RevId: 678da0fd32bc9314827bf2dd767c6d1728047566

Summary: Simple fix for examples to render properly and fix some docs formatting

Test Plan: rendered

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6093

GitOrigin-RevId: 79a85967f43d73a03d72d4defeb0b175e316ca71

Summary: we want to make it clear what the type of the suggestion is, and the state it's in

Test Plan:
screenshot:
{F92930}

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6083

GitOrigin-RevId: 883e42cb0f063ea7e2306b297b0825483ce3e95a

Summary:
made two copies of the bundle.json as named them as bundle-px.json and bundle-os.json for testing.
this just updates the UI so it fetches both px and open-source bundles in a Promise.all.

The pxl dev server should operate as normal, unless we also want the pxl dev server to host opensource scripts as well. that would just be a small modification to the current code.

Test Plan: ran webpack

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6085

GitOrigin-RevId: 5b3fb7f8034d5a97497d799084f42631c0f333bc

Summary: TSIA

Test Plan: N/A

Reviewers: philkuz, #engineering, zasgar

Reviewed By: #engineering, zasgar

Differential Revision: https://phab.corp.pixielabs.ai/D6096

GitOrigin-RevId: 32cce6277180ca1bdd5418f612b7436fd483ca81

Summary:
High CPU is caused by having a background SVG image. It's not clear exactly why, but
using the PNG is smaller and does not cause the CPU spike.

Test Plan: Tested on Chrome.

Reviewers: philkuz, michelle, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6022

GitOrigin-RevId: b6406f1dd813a779ffbf28e39930fb369169abf6

Summary: tsia

Test Plan: tested

Reviewers: zasgar, nserrino, #engineering

Reviewed By: nserrino, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6094

GitOrigin-RevId: f9d7311c4d01196df9ec3d15602c920a89f591b9

Summary:
This rule will return an error if someone calls df.stream() on a node with a blocking ancestor, or a node that has children besides sinks.
PP-2115 tracks the work to add things like streaming union, agg, join in order to remove these limitations.
Depends on D6080.

Test Plan: added unit test

Reviewers: philkuz, jamesbartlett, #engineering, zasgar

Reviewed By: #engineering, zasgar

JIRA Issues: PP-2161

Differential Revision: https://phab.corp.pixielabs.ai/D6087

GitOrigin-RevId: 0699ae9e1f67c50fc1efab3aa8f1026074a3d2a8

Summary:
Removes unused function.

Fixup

Test Plan: UI/Linter

Reviewers: michelle, nserrino, philkuz, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6092

GitOrigin-RevId: 9cee2ecbd3d8860846aeb6153afe6bbdb5e030b1

Summary:
If we get the exact same timestamp and it happens to sort so that parent > next_parent then the loop hangs forever.

In the longer term we should refactor to do this merge as a heap.

Test Plan: Existing

Reviewers: michelle, philkuz, nserrino, #engineering

Reviewed By: michelle, nserrino, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6078

GitOrigin-RevId: bc802d88fabca64adc0549a04bcffef8ce39cd74

PP-2117: ExecuteScript API should stream results as soon as they are available, and not batch them up in the qb.

Summary:
This completes the series of diffs that refactor the query broker to stream results using the query forwarder, rather than batching them up using the new deleted query executor before sending them back.
I also added new tests, since we didn't actually have any ExecuteScript tests before. Depends on D6073.

Test Plan: added.

Reviewers: michelle, philkuz, zasgar, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2117

Differential Revision: https://phab.corp.pixielabs.ai/D6074

GitOrigin-RevId: d494e7de693ddc747538e4802125d74c431f1446

Summary:
This is going to be the pxl way to mark that a query should be executed in "infinite" streaming mode.
Next up, a rule that turns StreamIR nodes into a modification on their parent MemorySources (setting stream=true).

Test Plan: added a unit test

Reviewers: philkuz, jamesbartlett, zasgar, #engineering

Reviewed By: jamesbartlett, #engineering

JIRA Issues: PP-2161

Differential Revision: https://phab.corp.pixielabs.ai/D6080

GitOrigin-RevId: 5cacfefec022888c7f2a7b9c44e28a8676aa3d02

Summary: Required as part of Apache2 license.

Test Plan: N/A

Reviewers: michelle, philkuz, nserrino, #engineering

Reviewed By: philkuz, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6088

GitOrigin-RevId: e1a89b2bd5a76fdf747971603baec1e1c326a5ee

PP-2161: Add streaming option to memory source IR node and propagate it down to the memory source plan node.

Summary: Next up is an analyzer rule that takes StreamIRs, and sets streaming=true on their parent memory sources.

Test Plan: added unit tests

Reviewers: philkuz, jamesbartlett, zasgar, #engineering

Reviewed By: philkuz, jamesbartlett, #engineering

JIRA Issues: PP-2161

Differential Revision: https://phab.corp.pixielabs.ai/D6082

GitOrigin-RevId: a63c45e6f97bb2763c89d1cd8b7a696e81af2402

Summary: More inline docs, part of PP-2147. Still have to parse these into something useful

Test Plan: runs without issues for now

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

JIRA Issues: PP-2171, PP-2172, PP-2147

Differential Revision: https://phab.corp.pixielabs.ai/D6076

GitOrigin-RevId: c9074875e15759b723dde6dcb5501d78048c3658

Summary: fix udf doc so that identity is different than description for args

Test Plan: updates tests

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6084

GitOrigin-RevId: 6210e55e34d471b259ff64733d8bef2079acb487

Summary: This is needed because we started bassing creds between PEM->Kelvin.

Test Plan: Tested e2e.

Reviewers: michelle, #engineering, nserrino

Reviewed By: #engineering, nserrino

Differential Revision: https://phab.corp.pixielabs.ai/D6079

GitOrigin-RevId: f78445785e8e3718db5b4560c4a59bacfe444221

Summary:
other diffs added the state to elastic and to the response proto. this diff actually pipes it from elastic to the response.
next diff will use the state to show an icon in the UI

Test Plan: ran plc-dev, confirmed that i see the state in the response

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6077

GitOrigin-RevId: 3ca1974ac75483ec533071928f9a18c83fc2f0d8

Summary: tsia. MemorySink switched over to looking at the table type but I didn't update GRPCSink to do the same.

Test Plan: added end to end tests

Reviewers: jamesbartlett, philkuz, zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6071

GitOrigin-RevId: 90d88311049b27cb2245252abfd47193837c59f8

PP-2117: Add helper functions and utilities in preparation to stream all agent queries to the client, rather than batching the results up

Summary: This was the result of me breaking two diffs up. The next one up will complete PP-2117, and have query results streamed directly from agent to client, rather than batched in the query broker.

Test Plan: added tests

Reviewers: michelle, philkuz, zasgar, #engineering

Reviewed By: michelle, #engineering

JIRA Issues: PP-2117

Differential Revision: https://phab.corp.pixielabs.ai/D6073

GitOrigin-RevId: e3aa30451c68b63c4cd7a11f67f6fb4fc0a143c6

Summary:
we want to show the state of each autocomplete entity so that users may know if they're looking at terminated/running pods/services/etc.
this diff adds the state to the proto. a followup diff should actually populate that field

Test Plan: unit test

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6061

GitOrigin-RevId: 80d18c817bc5b493034d0f831b0c69cb7be3faeb

Summary: we want to show the state when showing our results, so we need to include that in the data we index

Test Plan: unit test

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6063

GitOrigin-RevId: 7e117300cd03421b249b3fa33dd73da361b23cfa

Summary:
we should start on clustered (service) mode by default, especially since some cluster's pod map may be too much to render.
the component wasn't actually structured properly to start with clustered mode first, so there was a small refactor to make this possible

Test Plan: ran node_modules and checked that toggling the button works correctly

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6065

GitOrigin-RevId: a3d080eddf52cc2ede1670a212ce73e5eb0303fe

Summary:
Don't fail on containerd overlay paths. Just print warning and continue.

This at least enables kprobes to deploy properly.

Test Plan: Manual

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6068

GitOrigin-RevId: c817da2968cdb36ad8e1fda4371497b73959dca2

Summary:
We no longer use ReceiveAgentQueryResult in Carnot. This change updates the logic in Carnot and the agent manager to remove support for sending results over via that endpoint.
Depends on D6053.

Test Plan: existing

Reviewers: michelle, philkuz, jamesbartlett, zasgar, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2117

Differential Revision: https://phab.corp.pixielabs.ai/D6055

GitOrigin-RevId: edd6251a66e530cae0af4b1a6a43e9852f4bb937

Summary:
Not sure how to get around the lint error, because the service generated proto has an underscore in the package name too.
This will be used for unit tests in the query broker.

Test Plan: n/a

Reviewers: michelle, philkuz, zasgar, #engineering

Reviewed By: zasgar, #engineering

JIRA Issues: PP-2117

Differential Revision: https://phab.corp.pixielabs.ai/D6067

GitOrigin-RevId: d98d92065ed9f4deb5b24c09ed3edc19f89c2b0d

PP-2117: Introduce new QueryResultForwarder to support streaming Kelvin responses back from QueryBroker to client.

Summary:
This is the new QueryResultForwarder, which in the next diff will take over a lot of the work that the QueryExecutor used to do.
The QueryResultForwarder is built to handle streaming responses from the agent and forward those responses (transformed to the public proto)
to the client stream. Next diff will change the query broker server to integrate the new QueryResultForwarder.

Test Plan: added various tests, including cancellation, to check for hangs

Reviewers: michelle, zasgar, #engineering

Reviewed By: zasgar, #engineering

JIRA Issues: PP-2117

Differential Revision: https://phab.corp.pixielabs.ai/D6066

GitOrigin-RevId: ec1b6a82b63e713c45f8eb5fe49138c16c19482a

Summary: pipe it all around. untested have fun

Test Plan: untested, will test wednesday morn

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6069

GitOrigin-RevId: 542a37962227ee74b03d90ec3c6f7e06dd124d4e

Summary: LinuxKit uses /proc/config.gz, so required.

Test Plan: Basic test added. More testing forthcoming.

Reviewers: yzhao, #engineering, zasgar

Reviewed By: #engineering, zasgar

Differential Revision: https://phab.corp.pixielabs.ai/D6064

GitOrigin-RevId: 28206d29900c9fc329da26ca313e0b9a44f74b55

Summary: TSIA

Test Plan: N/A

Reviewers: philkuz, #engineering

Reviewed By: philkuz, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6062

GitOrigin-RevId: da6ea5bbf826c940dd8b62bacec5c351e9c67166

Summary: Moving to a <dst> = <src> data model.

Test Plan: Existing tests

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6046

GitOrigin-RevId: d50839d3fa803f5b79e42adf94fb76b49c54486f

Summary: streamable memory source. Basically always says it has batches remaining. A bit hacky before we figure out how the proto should look like.

Test Plan: add a test

Reviewers: nserrino, zasgar, michelle, #engineering

Reviewed By: nserrino, #engineering

JIRA Issues: PP-2119

Differential Revision: https://phab.corp.pixielabs.ai/D6058

GitOrigin-RevId: 3b4e66b2ab67cf9684cf77099afa30eee0da773f

Summary: Final major piece in being able to trace structs.

Test Plan: Test case in stirling_dt_bpf_test

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

JIRA Issues: PP-2143

Differential Revision: https://phab.corp.pixielabs.ai/D6040

GitOrigin-RevId: 985cc5b448e30361912688d253acc933e7e3cd67

Summary:
the current autocomplete doesnt show org-specific scripts. this is mostly because the script-autocompleting logic is temporary until we have an actual script indexer. for now, we are just using the bundle reader. a script in the bundle reader has an orgName for each script. however, we only can get the orgID from the jwt when autocompleting. one option is to add the orgName to the jwt, but that is redundant information.
another option is to make a request for the orgName, given the orgID, for every autocomplete call. but we don't want to slow down autocomplete with redundant requests.
instead, what I decided is to make a map of orgID -> orgName upon initialization of the API service. the suggester can then use this mapping to determine whether to show a script, given the orgID.
again, this is all temporary until we have script indexing.

Test Plan: ran plc-dev, checked that i could see pixielabs.org specific scripts

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6057

GitOrigin-RevId: 83c6385cec8a0c7d138cae7b9e8e11bfb0e746cd

Summary: Documentation.

Test Plan: None

Reviewers: philkuz, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2148

Differential Revision: https://phab.corp.pixielabs.ai/D6060

GitOrigin-RevId: 1b13cb9e0d1e63d9a871aced6cd10fb462b380d7