Summary:
We have seen multiple reports of our operator monitoring failing on
the k8s watcher and restarting the watcher also fails since the RV is too old.

On further investigation, it seems like we should be using informers to handle
resource watching, so switch to informers

Test Plan: skaffold deploy the operator with some additional logging

Reviewers: michelle, philkuz

Reviewed By: philkuz

JIRA Issues: PP-3066
Signed-off-by: Vihang Mehta <vihang@pixielabs.ai>

Differential Revision: https://phab.corp.pixielabs.ai/D9899

GitOrigin-RevId: 1f776f63db89e527d45d688d2d7d9be2b9a0eae0

Summary:
A lot of these blank imports were seemingly copy-pastaed.
vizier + operator work on many different kubernetes engines  and likely don't
need to rely on GCP auth.
We are also trying to ensure that pixie cloud works on a variety of environments
and should probably just import the entire auth package where needed.

Test Plan:
Ran a dev cloud + dev vizier + dev operator on my cluster.
Also tested a bazel built `px` cli

Reviewers: michelle, philkuz, zasgar

Reviewed By: zasgar

Differential Revision: https://phab.corp.pixielabs.ai/D9798

GitOrigin-RevId: fa747111015fb4c15e94fb7c3aa2e1e1e9d7fd84

Test Plan: Create a RC, update to it

Reviewers: michelle, philkuz

Reviewed By: michelle

JIRA Issues: PP-2993

Differential Revision: https://phab.corp.pixielabs.ai/D9747

GitOrigin-RevId: 4a11c87dbc3e2d59bc18fa05ada24edd3dc5cbcc

Summary:
Using `client-gen` requires apis to be in `<groupname>/<versionname>`
This is the folder structure used by most CRDs (I looked at etcd, jaeger, nats)
and so let's follow the same.

Test Plan: N/A. Moving files around.

Reviewers: michelle, philkuz

Reviewed By: michelle

Differential Revision: https://phab.corp.pixielabs.ai/D9693

GitOrigin-RevId: b693de36f511936ca6569509cb770da675470d76

Summary:
Switches over from poll to push just like the node watcher.
Note that status aggregation currently causes a fetch on the vizier CRD. There's
room for improvment in the design there.

Test Plan: Existing tests + skaffold vizier

Reviewers: michelle, philkuz

Reviewed By: philkuz

Differential Revision: https://phab.corp.pixielabs.ai/D9695

GitOrigin-RevId: 046018a63a5a2011c2d9459667e138a94a56fc72

Summary: Mostly cleaning up interfaces and making it easier to parse.

Test Plan: Existing test

Reviewers: michelle, philkuz

Reviewed By: michelle, philkuz

Differential Revision: https://phab.corp.pixielabs.ai/D9681

GitOrigin-RevId: de1f11838845b5ce8612655a8b030a8c9bb1d4b2

Summary:
this diff adds the nodeWatcher, which follows the potential new channel/status aggregation scheme discussed.
the nodeWatcher runs in a separate goroutine and separately tracks its status from the monitor itself. the status only updates when there is a node update, and the update is published to the monitor through a channel.
currently the monitor has a separate goroutine which loops through the updateChannels and tracks the last status for each.

Test Plan: deployed on skaffold, ran unit test

Reviewers: vihang, philkuz

Reviewed By: vihang

JIRA Issues: PP-2963

Differential Revision: https://phab.corp.pixielabs.ai/D9648

GitOrigin-RevId: 116652a3c20bd6faf85d83352d11e9e7587374ee

Summary: we do not plan to support viziers that are 1+ minor versions too old. when the user's vizier is too old, we should let them know they are in a degraded state and that they should not expect all functionality to work.

Test Plan: unit test

Reviewers: philkuz, vihang

Reviewed By: philkuz

JIRA Issues: PP-2961

Differential Revision: https://phab.corp.pixielabs.ai/D9636

GitOrigin-RevId: c936fa211c30bf494103f2d3a6be4907b1b89045

Summary: This uses the convenience methods from `k8s.io/client-go` and removes some of our code dupe.

Test Plan: skaffold

Reviewers: michelle, philkuz

Reviewed By: philkuz

Differential Revision: https://phab.corp.pixielabs.ai/D9625

GitOrigin-RevId: 46f5dd45927f3b4000e80a3c10c4d908c81a3286

Summary:
Add data pipeline, detector, and tests for metadata PVC. Currently checks to see if the PVC is stucking Pending mode because StorageClass doesn't exist.
Another option we had for this particular check was to look at events and regex match the occurring event, or check the reason. I found my chosen option easier to evaluate and more stable, but open to suggestions.

Only checks metadata pvc if etcd is not used.

Test Plan: Added tests for logic, test on skaffold by forcing a broken pvc

Reviewers: michelle, vihang, zasgar

Reviewed By: michelle

JIRA Issues: PP-2957

Differential Revision: https://phab.corp.pixielabs.ai/D9606

GitOrigin-RevId: 6399f3c02d8dea2c04087907e6d7a29d9133b7e4

Summary:
Cleaning up the operator monitoring code in preparation for knocking out PP-2874.
A summary of changes ->
1. Added some types that were previously missing (VizierPhase on api defs, new VizierReason type in vzstatus.go)
2. Separate out ReconcileStatus into two functions (1 - gets a reason from a specific subcomponent, 2 -converts the more specific reason to a Phase)
    * the Reason will be used for a healer
    * Showed proof of concept of different components providing data (GetCloudConnState).

Test Plan: Ran tests and they work, will try to deploy operator. Most worried about the VizierPhase type addition in the api defs.

Reviewers: michelle, vihang, zasgar

Reviewed By: michelle

JIRA Issues: PP-2874

Differential Revision: https://phab.corp.pixielabs.ai/D9589

GitOrigin-RevId: d3bac97662c050b583f6e71959663800f5e940dc

Summary:
the operator's monitor is responsible for:
1. watching the K8s API for tracking the latest pod statuses. we keep a version of this in memory since the # of vizier pods is small. the retry watcher code is mostly borrowed from the vizier metadata's retry watcher code.
2. periodically pinging the statusz endpoints of the vizier pods to determine the latest state. for now, this logic is very simple and we only look at whether the cloudConnector can run basic queries (this is the status quo). once the entire pipeline is more built out, we can make this even smarter.

Test Plan: skaffold deployed vizier and operator. played around with cloudconn status to make sure the CRD status gets updated to reflect its new state.

Reviewers: vihang, nserrino, zasgar

Reviewed By: vihang

JIRA Issues: PP-2875

Differential Revision: https://phab.corp.pixielabs.ai/D9466

GitOrigin-RevId: 8482820e17982959bb47615068bfdd98a6ddbb0b

Summary: Vizier Controller Connects to Cloud

Test Plan: Manual testing of vizier operator

Reviewers: nserrino, michelle

Reviewed By: michelle

JIRA Issues: PP-2812

Differential Revision: https://phab.corp.pixielabs.ai/D9161

GitOrigin-RevId: f4d9562d4621d1dcfcd52be19e3f9d828799e96d

Summary:
we used to rely on the cert-provisioner job to generate certs for a vizier when it starts up for the first time.
we can now do this in the operator. this code is mostly a copy from the current cert-provisioner job. once the operator migration is done, we can finally remove the cert provisioner job. for now, it will still start up, but be a no-op.

Test Plan: run operator, deploy new vizier CRD. confirm that vizier starts up properly

Reviewers: vihang, vjain, zasgar, nserrino

Reviewed By: vihang

JIRA Issues: PP-2793

Differential Revision: https://phab.corp.pixielabs.ai/D8870

GitOrigin-RevId: 0201df55b6db1f0bafac9d872a838dd66f25d91a

Summary:
before deploying vizier to a user's cluster, we need to update the configuration to add the user's custom labels, annotations, and resource requirements.
this diff adds the implementation for the functions which add the configurations to the k8s resources.
unfortunately, its not the prettiest since everything is an unstructured object consisting of maps of interfaces. most of the code is checking the map and casting. any suggestions for making this look nicer are appreciated...

`addKeyValueMapToResource` is an exact copy of code that we had before we had templates.

Test Plan: deployed CRD with operator running. confirm that the deployed vizier pods had the annotations and resource requirements

Reviewers: vihang, zasgar, nserrino

Reviewed By: vihang

JIRA Issues: PP-2653

Differential Revision: https://phab.corp.pixielabs.ai/D8823

GitOrigin-RevId: eca0c501ace2e2b3c6d8225d148247668947a579

Summary:
This diff implements the Vizier deployment flow in the operator.
there are two main components still missing from this flow:
1. Marking the vizier status as "running" once everything has started up and is healthy. we can borrow from the CLI deploy logic for this.
2. Populating the vizier YAMLs with custom annotations/labels and resource requirements.
these changes will come in a follow up diff.

with the changes in this diff, the operator properly deploys a running version of Vizier when the CRD applied

Test Plan: ran skaffold, deployed CRD, watch namespace to ensure vizier starts up properly

Reviewers: zasgar, vihang, nserrino

Reviewed By: vihang

JIRA Issues: PP-2653

Differential Revision: https://phab.corp.pixielabs.ai/D8820

GitOrigin-RevId: 2c47d701fdbb03b35d29b0a92f7f65a627b37b3d

Summary:
this sets up the basic flow in the operator's reconcile phase. following diffs will actually implement each of these functions. the implementation of these functions will be heavily borrowed from our current deploy/update/delete in the CLI/updater job.
1. deletion: if the reconciler was triggered with a CRD that no longer exists
2. creation: if the reconciler was triggered with a CRD that has no known status.
3. update: if the reconciler is triggered with a CRD that does have a known status/deployed vizier.

Test Plan: ran skaffold, created a CRD with no version

Reviewers: vihang, zasgar, nserrino

Reviewed By: vihang

JIRA Issues: PP-2653

Differential Revision: https://phab.corp.pixielabs.ai/D8809

GitOrigin-RevId: d35e5286c599b5fe5d7907f70517dc9544b65f2f

Summary:
Adding initial boilerplate for our Vizier operator, based off of the generated kubebuilder project.
kubebuilder generated:
- contents of operator directory
- k8s/operator/crd
I heavily modified the contents of the operator directory to remove the additional items we didnt need and made it work with our build system. removed most of the items in the Makefile except for the commands that help generate the k8s crd yaml and generated go files.

Test Plan: deployed the crd, deployed the operator, and deployed an instance of the Vizier custom resource. verified that the operator detected the change and triggered the "Reconcile" function.

Reviewers: zasgar, vihang, #third_party_approvers

Reviewed By: vihang, #third_party_approvers

JIRA Issues: PP-2650

Differential Revision: https://phab.corp.pixielabs.ai/D8367

GitOrigin-RevId: 2c0ae0e45c97309258f083eea5d33acb2fc15578

Summary: This adds the license headers to all our remaining files.

Test Plan: manual inspection

Reviewers: michelle, vihang, oazizi, #third_party_approvers

Reviewed By: vihang, #third_party_approvers

JIRA Issues: PP-2565

Differential Revision: https://phab.corp.pixielabs.ai/D8251

GitOrigin-RevId: 7913fa60b6a73da77ce6de33a44bb57e2baac775

Summary: Removes pixielabs from the repo name.

Test Plan: existing

Reviewers: michelle, oazizi, vihang, #third_party_approvers

Reviewed By: michelle, #third_party_approvers

JIRA Issues: PP-2609

Differential Revision: https://phab.corp.pixielabs.ai/D8196

GitOrigin-RevId: d18846e6bedc0ebfabdf45c392af32707d72c380

Summary:
Move from `visibility:public` to `src:__subpackages__`
Further restrict `src/stirling`, `src/cloud` and `src/vizier` to their own
directories where possible.

Test Plan: Build and test all.

Reviewers: michelle, zasgar, oazizi

Reviewed By: zasgar

JIRA Issues: PP-2567

Differential Revision: https://phab.corp.pixielabs.ai/D8031

GitOrigin-RevId: a9547b64db95578d9e490bcdf6e2a8c1920100ee

Summary:
This is much cleaner than the earlier approach we had to adopt because of limitation in gazelle. There are a
few missed cases and some bad package naming. Will address those shortly.

Test Plan: existing.

Reviewers: michelle, vihang, #third_party_approvers

Reviewed By: vihang, #third_party_approvers

Differential Revision: https://phab.corp.pixielabs.ai/D8018

GitOrigin-RevId: 29cffda55ef72043830d93399aabf69a685bb149

Summary: It's more than a util now.

Test Plan: existing

Reviewers: michelle, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D7033

GitOrigin-RevId: 79fbc746ad7f935b7793b15434e5b46e8a5cc205

Summary: Report failures to sentry.

Test Plan: Tested manually.

Reviewers: #engineering, philkuz, michelle

Reviewed By: #engineering, philkuz

Differential Revision: https://phab.corp.pixielabs.ai/D3830

GitOrigin-RevId: 80741a8ddaf06af7012f2d48fded57c9aa4fa91c

Summary: Adds segment analytics to the CLI. Also introduces config to the CLI that we can use to track other things like cloud_addr, etc.

Test Plan: Tested manually.

Reviewers: michelle, #engineering

Reviewed By: michelle, #engineering

JIRA Issues: PL-1549

Differential Revision: https://phab.corp.pixielabs.ai/D3758

GitOrigin-RevId: 82de5c34348c320ff4101f9785e3ea098b977e45

Summary: Restructuring to match typical structures for open source CLI tools.

Test Plan: N/A

Reviewers: michelle, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D2152

GitOrigin-RevId: 71a3791ae91acaa6e88237939a9bb967a77601fb

Summary: CLI fetches auth tokens and saves them in ~/.pixie/auth.json. We can use this to fetch resources from the cloud.

Test Plan: Manually tested.

Reviewers: michelle, #engineering

Reviewed By: michelle, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D2117

GitOrigin-RevId: 67353407f2a058a1cfe01f53b9f68f9ef26f83c2

Summary: Makes things more consistent.

Test Plan: Existing

Reviewers: michelle, #engineering, oazizi

Reviewed By: #engineering, oazizi

Differential Revision: https://phab.corp.pixielabs.ai/D2013

GitOrigin-RevId: 7962522cb4a1428ed801af1f4c4d8e73673b1788

Summary: Before you had to run it as two separate steps. Instead, deploy should also run install certs.

Test Plan: tried it manually

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D1965

GitOrigin-RevId: 49fdedd992565e9f285db4f9165c5b5c14bb1caf

Summary:
We need a way to generate certs on all of our customer's clusters.
This script creates a CA (or loads a CA from a given path), and generates the server and client crts/keys.

Test Plan: generated certs using the script, loaded them as secrets into minikube, ran skaffold to make sure all of our services could talk to each other.

Reviewers: zasgar, #engineering

Reviewed By: zasgar, #engineering

Differential Revision: https://phab.corp.pixielabs.ai/D1493

GitOrigin-RevId: 1ee51bb4c95dc32ce67da5e21385f6636b3ce2f4