Summary:
Reads `go.sum` to check all golang deps. Tries to fetch license from `github.com` and if it fails, tries `pkg.go.dev`.
Reads `.gitmodules` for submodules and tries to fetch licenses from `github.com`.
Uses the npm `license-checker` package to fetch licenses for npm deps.
Reads `repositories.bzl` for other deps and tries to fetch licenses from `github.com`.

Falls back to reading a manual JSON input for deps that we couldn't automatically fetch licenses for.

Test Plan: Run the license fetcher script.

Reviewers: zasgar, philkuz, #engineering, michelle

Reviewed By: philkuz, #engineering, michelle

Subscribers: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D6652

GitOrigin-RevId: a7afed22cc3d65c4a87fd161f0fb1ce5ef619e60

Summary: Cleaning up the cluttered directory structure a bit.

Test Plan: Existing tests

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6481

GitOrigin-RevId: 1e34e3ee14ef5600cd67328176694e5387095714

Summary:
We ran into this issue in both customer prod clusters, where the cache was unable to flush because we kept trying to flush an entry that was larger than the maximum request size that we can send over etcd.
we already try to batch etcd operations so that the request is not larger than the maximum request size. however, this single operation is already larger than the max request we can send.
i think we need more information to determine how to handle this case.
- if it is a k8s update, which we got directly from the k8s api, we should consider increasing the maximum request size that etcd permits, since most likely we don't want to cut out any information in this message.
- if it is our own entry (agents, schemas, etc), then we should consider looking into ways we can shrink the message or split it into separate entries.
to figure this out, i just added a log for the next time the error occurs.

Test Plan: n/a

Reviewers: nserrino, zasgar, #engineering

Reviewed By: nserrino, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D6329

GitOrigin-RevId: 679725b32c2818cc5634bedfdb91d4f526c2c770

Summary:
Adds a UDF for the transformer model
Adds a UDF for the sentencepiece model
Adds a UDA for fitting kmeans
Adds a UDF for taking fitted kmeans centroids, and mapping to cluster ids.

Test Plan: Added tests, and also tried end2end

Reviewers: #engineering, zasgar

Reviewed By: #engineering, zasgar

Subscribers: philkuz, zasgar

Differential Revision: https://phab.corp.pixielabs.ai/D6295

GitOrigin-RevId: 81c1a0da8b5dbd7051c11caf5caafae5568e70bb

Summary:
This adds the ModelPool which allows UDFs to borrow a pointer to a pre-initialized model. Right now, the pool will only create one model, and allow calls must share that copy, but this can be changed in the future if we need more parallel queries.

This diff also implements inference for the transformer style model.

@zasgar We talked about this awhile back but I think I strayed a bit from what we were talking about, so would be good to hear your thoughts on this one.

Test Plan: Added some tests and used in end2end

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Subscribers: zasgar

Differential Revision: https://phab.corp.pixielabs.ai/D6294

GitOrigin-RevId: f34c22892644f413025bf4e7973b5a8e25f3d07a

Summary: Protobuf will be used as part of Pxl autodoc. This protobuf will act as the holder both of messages from cpp -> GO docstring parser and go to UI  docs renderer

Test Plan: n/a

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D5962

GitOrigin-RevId: f012db381efcc2c8661af79583e5d4a58c1158de

Summary: Will be used for testing changes to the query broker, specifically the implementation of the API TransferResultChunk.

Test Plan: n/a

Reviewers: michelle, zasgar, philkuz, #engineering

Reviewed By: philkuz, #engineering

JIRA Issues: PP-2117

Differential Revision: https://phab.corp.pixielabs.ai/D5940

GitOrigin-RevId: 73a7c91abf5e17314db948014b43f55c4bbf2a72

Summary: tsia

Test Plan: everything should build

Reviewers: zasgar, oazizi, yzhao, #engineering

Reviewed By: oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D5817

GitOrigin-RevId: 9f97fe56a6a0d87df5845452b11a57eb433be5f4

Summary: tsia

Test Plan: n/a

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D5779

GitOrigin-RevId: 6b92f48f4eb2c22414edcfa3f7739a7cbd5d8522

Summary:
this diff adds the probe manager to MDS. it is currently not hooked up yet--that is to come in a later diff.
the probe manager is responsible for handling any probe creation/updates and sending off registration messages to agents over NATS.

the plan is to have a reference to the probe manager in metadata/server.go, which receives RPC requests creating/checking probe status.
there will also be a reference to probe manager in the agent topic listener, where we receive probe update messages.

Test Plan: unit tests

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D5704

GitOrigin-RevId: e623f420e4620e5d403122641a7f6e12bf8bdc2a

Summary: tsia

Test Plan: ran update_go_protos, make gazelle

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D5696

GitOrigin-RevId: e45d6c0ff6ace06b74ac8a8a386412e429be0a75

Summary:
This adds the proto format for how we plan to store ProbeInfo in etcd in MDS.
Currently, MDS stores protos from agentpb and metadatapb, which are all part of shared directories and used in a couple other places.
ProbeInfo is the only proto which is used internally in MDS, so I have created a new file for such protos.

the probe message just contains the ID, and the list of agents who have failed/suceeded to register the probe.

Test Plan: n/a

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D5671

GitOrigin-RevId: 4dcb89c1d192c3a76f2de5864ab66cba438e2c50

Summary:
this diff adds the RegisterProbe messages for the QB->MDS->PEM messaging pipeline.
although the contents of the messages are currently the same, I created two separate messages for QB->MDS and MDS->PEM.
that is because the messages from QB->MDS is intended as a message to register probes on all agents, whereas the message from MDS->PEM is the intent to register probes on a single agent.

Test Plan: existing tests should pass

Reviewers: zasgar, oazizi, philkuz, yzhao, #engineering

Reviewed By: zasgar, oazizi, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D5664

GitOrigin-RevId: ab1636328693d93611fccf19ceca3ad47b7128aa

Summary: TSIA

Test Plan: Ran skaffold

Reviewers: philkuz, zasgar, #engineering

Reviewed By: philkuz, zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D5580

GitOrigin-RevId: 104ced2bf27c4926d04c39491c4bae089e4c74c7

Summary: This adds pre-packaged timeconst.h files to accompany the prepackaged linux headers.

Test Plan: Tested on EC2.

Reviewers: yzhao, zasgar, #engineering

Reviewed By: yzhao, zasgar, #engineering

JIRA Issues: PP-1944

Differential Revision: https://phab.corp.pixielabs.ai/D4926

GitOrigin-RevId: 101c7c2ead324fe0042776c2ddcbe88726eda4a4

Summary:
we recently updated grpc, which slightly changes how it uses Addresses (https://godoc.org/google.golang.org/grpc/resolver#Address)
See: ServerName. it says that if serverName is not empty, that is the name it will use to authenticate with the TLS cert. kuberesolver, the package we're using to resolve kube addresses does not leave this field empty, but rather uses the pod-name, which fails the TLS check. instead, it should be setting this field as "".  patched the kuberesolver library to do that.

Test Plan: ran cloud dev and made sure could login

Reviewers: zasgar, philkuz, #engineering, nserrino

Reviewed By: #engineering, nserrino

Differential Revision: https://phab.corp.pixielabs.ai/D4890

GitOrigin-RevId: fd5f2395c3a2fa5dc22e2d04c32520fe4539d1c0

Summary:
Depends on D4477. We are constructing bloom filters that contain all of the k8s entities
that a given agent has stored on it. Those bloom filters will be created by the agents, sent to the
metadata service with the agent heartbeats, and queried by the query broker when a query is executed.
This diff creates a place for the wrapper of those bloom filters, AgentDataInfo, in the metadata store.
AgentDataInfo will eventually store more than just information about K8s metadata, it will also store
things like which columns in an agent table contain which categorical values, etc.

Test Plan: added

Reviewers: michelle, zasgar, philkuz, #engineering

Reviewed By: michelle, #engineering

JIRA Issues: PP-1809

Differential Revision: https://phab.corp.pixielabs.ai/D4651

GitOrigin-RevId: a49b14095d97f76b1fb0a5941132b44f611302b0

Summary:
We now include compressed Linux header packages, and decompress the closest match if we can't locate host headers.

Before this work started, we had included a single linux headers package consuming aboud 230 MB.

Now, we include 12 sets of linux header packages, but compressed. In total, they consume ~96 MB.

So we've saved space, and we've made things more robust.

Only downside is some initialization time is used to decompress the files, but this is minor. And it is only incurred when host header packages are not located.

Test Plan: Updated some tests, and tested manually.

Reviewers: yzhao, zasgar, #engineering

Reviewed By: yzhao, #engineering

JIRA Issues: PP-1863

Differential Revision: https://phab.corp.pixielabs.ai/D4635

GitOrigin-RevId: 9fae1e5a80a4772b354ac37a25902c31a6dd271d

Summary:
This removes a bunch of unneeded files from the pre-packed headers.

With the space savings, more linux headers are included in the base image.

This should make us more reliable across Linux versions.

Test Plan: Manual.

Reviewers: #engineering, zasgar, yzhao

Reviewed By: #engineering, zasgar

JIRA Issues: PP-1864, PP-1863

Differential Revision: https://phab.corp.pixielabs.ai/D4622

GitOrigin-RevId: ffb9cc3aea44b678dcff594b7edec56430e522a3

Summary:
This diff adds multiple pre-packaged linux headers into the PEM/stirling base image.

Stirling then chooses the closest Linux kernel match in case it can't find the host headers.

Test Plan: Some tests added, and tested manually too.

Reviewers: yzhao, zasgar, #engineering

Reviewed By: zasgar, #engineering

JIRA Issues: PP-1863

Differential Revision: https://phab.corp.pixielabs.ai/D4605

GitOrigin-RevId: 02fdf0332ce283ac8d31cff06d5645cb48ec98f2

Summary:
We suspect a mis-matched header may be causing Stirling issues when it has to use the prepackaged headers.

To address this, this includes a set of Linux headers build from version 5.3.18.

The headers aren't used yet. This just updates the generation code, and makes the tarball accessible.

Test Plan: Ran the script to build the headers. They look fine.

Reviewers: yzhao, #engineering, zasgar

Reviewed By: #engineering, zasgar

JIRA Issues: PP-1789

Differential Revision: https://phab.corp.pixielabs.ai/D4583

GitOrigin-RevId: efabc0bf50e37344f79e53da648d90c0db94cd02

Summary:
A better place for our tracing targets to live. These simple apps are used by both experimental probes and Stirling tests.

In sum
//experimental/bpf/tracers/bin -> //demo/client_server_apps
//demos/perf/bcc -> //experimental/bcc_perf

Test Plan: Jenkins

Reviewers: yzhao, zasgar, #engineering

Reviewed By: yzhao, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D4560

GitOrigin-RevId: 7ee5a9ad28d384cefcd61352510ef67aa1befa9d

Summary:
tsia

add vispb to top level BUILD.bazel

Test Plan: edited existing

Reviewers: jamesbartlett, michelle, philkuz, zasgar, #engineering

Reviewed By: zasgar, #engineering

JIRA Issues: PC-94

Differential Revision: https://phab.corp.pixielabs.ai/D4379

GitOrigin-RevId: 61aa8be50258cc7975283053cd2f459f2f05d6a9

Summary:
This provides the stimulus that exposes a gap in our HTTP2/GRPC uprobes.

Putting into experimental for now, but will likely move to Stirling as part of a test, once the uprobes are fixed.

Depends on D4337.

Test Plan: Manually run.

Reviewers: yzhao, #engineering

Reviewed By: yzhao, #engineering

JIRA Issues: PP-1751

Differential Revision: https://phab.corp.pixielabs.ai/D4340

GitOrigin-RevId: 90df45aa0099c583cff312dde7ca20d0cbcb36cb

Summary: This adds build avoidance by using bazel query to make sure we only run necessary targets.

Test Plan:
Tested with CC change.
Tested with Go Change.

Reviewers: michelle, oazizi, #engineering

Reviewed By: oazizi, #engineering

JIRA Issues: PP-1766

Differential Revision: https://phab.corp.pixielabs.ai/D4266

GitOrigin-RevId: d7f681e94a470f6a95bb99e63ebb8ce563f70b64

Summary: as discussed, we no longer want a separate autocomplete service. instead, we will be handling autocomplete in the API service. however, we will still pull out the autocomplete logic in a separate module (the old autocomplete folder we had before).

Test Plan: n/a

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D4214

GitOrigin-RevId: a89d0143e8889e995d8ecc1e4559a0fa92c1e4ab

Summary: we are adding an autocomplete service. this diff contains the boilerplate code necessary for getting that service up and running. it currently does nothing

Test Plan: ran in skaffold, made sure service is up and running

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D4147

GitOrigin-RevId: 0804f58d7cb3f12eccd5b938c4ebe209c109fffe

Summary: Depends on D3948. First in a series of refactors to repurpose Flags for the new setup.

Test Plan: All tests pass, no changes other than a rename

Reviewers: nserrino, jamesbartlett, #engineering

Reviewed By: nserrino, #engineering

JIRA Issues: PL-1630

Differential Revision: https://phab.corp.pixielabs.ai/D3964

GitOrigin-RevId: 9492984e8863f5a44ebf8c8138843ee80ed5c7f6

Summary: Depends on D3740.

Test Plan: existing, we also need to manually test this with the corresponding ui changes

Reviewers: #engineering, zasgar, michelle

Reviewed By: #engineering, michelle

JIRA Issues: PL-1553

Differential Revision: https://phab.corp.pixielabs.ai/D3770

GitOrigin-RevId: 599e33bacd9cef0786c3d5795a91849179cec4c3

Summary:
Depends on D3910.
Adds shared scriptspb proto definitions.
These will be used both by the compiler and by the scriptmgr in the cloud so they need to be in shared.

Test Plan: N/A

Reviewers: #engineering, philkuz

Reviewed By: #engineering, philkuz

Differential Revision: https://phab.c.pixielabs.ai/D3916

GitOrigin-RevId: 86ddf3f35847e0141bc15d830d7a2fcfb6360eee

Summary: when vzmgr receives a vizier connected request, it should broadcast to NATS. for now, I just publish to the "VizierConnected" channel with the vizier's id, org id, and current resource version

Test Plan: unit test

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D3717

GitOrigin-RevId: 6b82f02660c3783433757d52353b403a9d68d727

Summary: This makes the protobufs more consistent.

Test Plan: Existing tests.

Reviewers: michelle, philkuz, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D3697

GitOrigin-RevId: df2b17130cf7963391516107b213d6d3e115197a

Summary: Encapsulate all top level messages between vizier and cloud.

Test Plan: N/A

Reviewers: michelle, philkuz, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D3696

GitOrigin-RevId: b5bedc840b4663a7b2c7f6dd69c1e454e03a1951

Summary:
We no longer need this since it'll be handled by the NATS GRPC bridge.

Depends on D3692.

Test Plan: N/A

Reviewers: michelle, nserrino, jamesbartlett, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D3693

GitOrigin-RevId: c84bcf13f0b180446d4b4ad2484f4c30a25b7325

Summary:
The metadata service should have a rpc stream for getting metadata updates.
originally i said I would split this into two calls, one for "syncing" and one for subscribing to updates, but i realized just doing it in a single call makes more sense and matches  how the k8s api watcher works
the cloud connector should send a GetMetadataUpdatesRequest with the resourceVersion it wants. metadata service is this responsible for getting all updates currently existing since that resource version + sending future resource updates

Test Plan: n/a

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D3606

GitOrigin-RevId: c6453a2280b713c6f3657cde36f7abb526953c91

Summary: we need to translate between the pbs we're using internally to the new vizierpb format. these are just the utils to do so, the code that actually applies the translation will be in a followup diff.

Test Plan: unit test, tested when running skaffold

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D3572

GitOrigin-RevId: 7c0c522ff20ae22e8b487fe2d02df917261b22db

Summary: Depends on D3525. Follow up diff will have a rename of the namespaces.

Test Plan: tests pass, just a refactor

Reviewers: #engineering, zasgar, jamesbartlett, nserrino

Reviewed By: jamesbartlett

JIRA Issues: PL-1473

Differential Revision: https://phab.corp.pixielabs.ai/D3526

GitOrigin-RevId: 610a06247db629d186b945ed520e20a22fc2b139

Summary: Need new proto messages for the Query Flags and also wanted to add flags to QueryResult . Need query result message in the planner, will update in the following diff with that change.

Test Plan: tests pass, this is just a refactor + new definition

Reviewers: nserrino, jamesbartlett, michelle, zasgar, #engineering

Reviewed By: nserrino, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D3371

GitOrigin-RevId: 0c47471eb11223299c84d4b43d0adf0779420908

Summary:
Make UDFInfo a part of the LogicalPlanner constructor and make it so that logical planner inits a registry_info from the udfinfo. This makes it so we can create a Logical Planner with the registry info and use that registry info for the duration of the compiler state object's lifetime.

Also updated cgo boundary so that we pass UDFInfo (serialized) into the constructor of the planner.

Test Plan: Tests should all work as before.

Reviewers: nserrino, zasgar, michelle, #engineering

Reviewed By: nserrino, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D3114

GitOrigin-RevId: 215a3880afc69e89c4cb13634e5d4b21c4370c5d

Summary:
In D3178, I accidently landed from the wrong machine and submitted the stale change.
This is the patch to get it to the correct version.

Test Plan:
  `bazel test --config=libc++ --keep_going //...`
  `bazel test --config=clang --keep_going //...`
  `bazel test --config=gcc --keep_going //...`

Reviewers: #engineering, michelle, oazizi, philkuz

Reviewed By: #engineering, oazizi

Differential Revision: https://phab.corp.pixielabs.ai/D3192

GitOrigin-RevId: e9092fdf0754c157778f8edafd98acca2393c2cb