This project is mirrored from https://gitee.com/cowcomic/pixie.git.
Pull mirroring failed .
Repository mirroring has been paused due to too many failed attempts. It can be resumed by a project maintainer.
Repository mirroring has been paused due to too many failed attempts. It can be resumed by a project maintainer.
- 18 Feb, 2022 4 commits
-
-
James Bartlett authored
Summary: Adds a `RecordGenerator` that generates postgres data. Blocked by D10797 Test Plan: No production changes. Tested that the benchmark outputs all the input records for the new postgres data. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi JIRA Issues: PP-3211 Signed-off-by:
James Bartlett <jamesbartlett@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10798 GitOrigin-RevId: 5594ea97de0e8815af417d09fe1f8386b6981271
-
James Bartlett authored
Summary: Adds two kinds of `PosGenerator`'s that generate "gappy" data (i.e. simulate data from BPF that has large gaps in the data stream): - One generates data that has gaps between simulated polling iterations. - One generates data that has gaps within a single polling iteration. Blocked by D10793 Test Plan: No production changes. Tested that the new PosGenerator's work with the benchmark. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi JIRA Issues: PP-3211 Signed-off-by:
-
Pete Stevenson authored
Summary: We included <utility> based on a complaint from our linter, but the code subsequently changed leaving a stale include. Here, we remove that. Test Plan: Build process. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi JIRA Issues: PP-2982 Signed-off-by:
Pete Stevenson <jps@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10811 GitOrigin-RevId: 8d74f0747abe2daea37e634db28863ae25837d21
-
James Bartlett authored
Summary: When we moved the retention size limit restriction from DataStream::ProcessBytesToFrames into CleanupEvents, we neglected to also move the ShrinkToFit so that the DSB would be shrunk to the retention size. This diff just moves the ShrinkToFit into CleanupEvents. No need to still call ShrinkToFit in ProcessBytesToFrames because CleanupEvents will be called briefly after the end of ProcessBytesToFrames no matter what. Test Plan: Existing tests pass. Tried it out in skaffold and made sure scripts work as normal. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Signed-off-by:
-
- 17 Feb, 2022 13 commits
-
-
Phillip Kuznetsov authored
Summary: Simple fix. Something I spotted while documenting the data pipelines. Test Plan: Unit tests pass Reviewers: michelle, nserrino Reviewed By: michelle Signed-off-by:
Phillip Kuznetsov <pkuznetsov@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10814 GitOrigin-RevId: b33a01cf0024d2dc21b02e71d9f10a02eb7d16ab
-
Omid Azizi authored
Summary: Update the Mux protocol ByteSize() function to be more accurate. Test Plan: TBD. Reviewers: #stirling, rcheng Reviewed By: #stirling, rcheng Subscribers: rcheng Signed-off-by:
Omid Azizi <oazizi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10745 GitOrigin-RevId: 0ae374ae50b05e71b5c3a6b3dbca5b042e4a33b9
-
Omid Azizi authored
Summary: Use more inclusive language. Test Plan: This is just a name change. Rely on existing tests. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Signed-off-by:
-
James Bartlett authored
Summary: - Added a data generation framework, that separates the generation of the data from the generation of the `pos` byte label for each event. Users of the data generation framework specify a `RecordGenerator` and a `PosGenerator`. The `RecordGenerator` has a single method `Next` that should return a record (a collection of traffic direction, frame pairs) that represents a single record that would be pushed to the TableStore if successfully parsed. The `PosGenerator` has a method that returns the byte position for the next event given the next event's size, and a method to signal the next polling iteration of data (useful for adding a gap in byte position between simulated polling iteration). Using the provided `PosGenerator` and `RecordGenerator` the data generation framework creates events per connection per simulated polling iteration (RecordGenerators are instantiated per connection and PosGenerators are instantiated per direction per connection). - To keep this diff as small as possible I only added a minimal set of Pos and Record generators in order to benchmark only HTTP1 traffic with continuous `pos` labels. Future diffs add more interesting PosGenerators as well as simple RecordGenerators for a variety of protocols. - Added the benchmark itself. The benchmark first generates all the necessary data using the above framework (as specified by `BenchmarkDataGenerationSpec`). Then during a single benchmark iteration, it will for each simulated "polling" iteration: push each event for that iteration to the SocketTraceConnector as if it had come from BPF (eg. via HandleDataEvent), then call `TransferData` once. During the first benchmark iteration the `MemoryTracker` will be enabled to keep track of starting, ending and peak memory usage for that particular benchmark. Finally after all benchmark iterations, the benchmark reports a wide variety of statistics. The statistics reported are: `PollIters` (the number of simulated polling iterations), `AllocPeak` (the maximum memory used during the benchmark minus the memory used before the benchmark started processing data), `PhysMem{Start,End}` (The physical memory used at the start/end of the benchmark, physical memory includes memory that the program has freed to the allocator but the allocator hasn't released to the OS), `AllocMem{Start,End}` (The memory that has not been freed to the allocator at the start/end of the benchmark), `EquivalendWorkloadThroughput` (The throughput in B/s of a workload that would produce similar load to this benchmark), `Bytes{Input/Output}`/`Records{Input/Output}` (The amount of bytes/records of data input/output to/from the benchmark, note that records input should equal records output if no data was lost), and finally `NumEvents` (the total number of events processed). - Because there are so many statistics, I added a variety of flags to control what statistics get printed, by default only `AllocPeak` and `PollIters` are displayed. Blocked by D10786, D10784, D10750 Test Plan: No production changes, tested that the new benchmarking code runs the benchmark successfully, and that for each protocol the generated data can be successfully parsed. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: yzhao, oazizi JIRA Issues: PP-3211 Signed-off-by: -
Natalie Serrino authored
Summary: I tried to do a px.abs on the result of px.parse_duration, but noticed that it didn't preserve the DURATION_NS type. This diff adds the correct semantic type to the result of px.abs. Test Plan: not needed, we use these same rules elsewhere, just need to add them here Reviewers: philkuz, jamesbartlett Reviewed By: jamesbartlett Signed-off-by:
Natalie Serrino <nserrino@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10809 GitOrigin-RevId: d7a22eac106a04fa7ac9c806457b594f18b1887d
-
yzhao1012 authored
Summary: This allows skaffold to change PEM source connectors without making code changes Test Plan: Tested with skaffold Reviewers: #stirling, jps Reviewed By: #stirling, jps Subscribers: oazizi Signed-off-by:
yzhao1012 <yzhao@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10794 GitOrigin-RevId: 637ba96e276b1a5c3edecd1d16a4e36569fb9400
-
yzhao1012 authored
Summary: One example of improving organization. Similar headers, like system_headers.{cc,h} are already in src/stirling/utils. Test Plan: Move code files, Jenkins build covers this Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Signed-off-by: -
James Bartlett authored
Summary: For the purposes of the benchmark, we want to be able to generate a bunch of wire data for each protocol. This diff adds the utilities to make data generation easier. These utilities could also be used in the tests instead of the ad-hoc char[] byte buffers that the tests currently used. This diff adds a couple examples in the tests to demonstrate this usage. Note, this diff also restructures the build file so that test utilities don't get built into the main binary. Test Plan: No functional changes, so relying on existing tests. This diff also increases test coverage. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi, yzhao JIRA Issues: PP-3211 Signed-off-by:
-
Omid Azizi authored
Summary: The old implementation of DNS' ByteSize() used a const_cast which can be avoided. This diff cleans that up. Test Plan: No functional change, so existing tests. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao Signed-off-by:
-
James Bartlett authored
Summary: For the purposes of the socket tracer benchmark, we want to be able to generate a bunch of wire data for each protocol. This diff adds the utilities to make data generation easier for pgsql. These utilities could also be used in the tests instead of the ad-hoc char[] byte buffers that the tests currently used. I added a couple examples in the tests to demonstrate this usage. This diff also restructures the build file, such that the test data is not compiled with the main cc library. Test Plan: No functional changes, relying on existing tests. Note that this diff increases test coverage. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao JIRA Issues: PP-3211 Signed-off-by:
-
Vihang Mehta authored
Summary: Use POST message type and JSON parse params. Test Plan: `curl -X POST -vvv --raw -H 'Content-Type: application/json' --data '{"chunk_size":10, "body_size":100}' http://localhost:8111/` Reviewers: zasgar, nserrino Reviewed By: zasgar Signed-off-by:Vihang Mehta <vihang@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10804 GitOrigin-RevId: fea928b276a065840723681a097042a5696a11a5
-
James Bartlett authored
Summary: This util will be useful for generating network packets for benchmarking purposes. Test Plan: Added unit test for new utility. Utility is currently unused so no need for more testing than that. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi Signed-off-by:
-
Zain Asgar authored
Summary: This is the first pass at the client for HTTP requests only. I'll do some reorganization of the directories after this lands for more consistency. Test Plan: Run manually ```./client seq http --addr http://xyz:8080/seq_test -n 100000 -c 50 -s 2000000 -r 1024` Reviewers: vihang, michelle, nserrino Reviewed By: nserrino JIRA Issues: PP-3222 Signed-off-by:
Zain Asgar <zasgar@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10796 GitOrigin-RevId: 14f490a87cc3e1674b972fb2dd594b2a00e68c82
-
- 16 Feb, 2022 18 commits
-
-
Pete Stevenson authored
PerfProfiler: collapse Java interpreter stackframes, i.e. such that deep Java stack traces are rendered without redundant Interpreter frames. Summary: Java stack traces are deep and often include multiple "interpreter" stack frames. To provide a better visualization of the data (and as a side effect, save on memory consumption by stack trace strings) we collapse repeated interpreter frames into one frame. Test Plan: Existing. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao JIRA Issues: PP-2982, PP-3235 Signed-off-by:
-
Pete Stevenson authored
PerfProfiler: on exit, or target proc. termination, remove all symbolization artifacts create inside the target container. Summary: TSIA Test Plan: Tested on local dev. cluster. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi JIRA Issues: PP-2982, PP-3252 Signed-off-by:
-
Pete Stevenson authored
Summary: This is prep. work for cleaning up the Java symbolizer (either when a target process dies or when Stirling gracefully exits). We want to remove the symbolization artifacts, but for that, we need to remember the host resolved path to the same. Test Plan: Existing. Reviewers: #stirling, yzhao Reviewed By: #stirling, yzhao Subscribers: yzhao JIRA Issues: PP-2982, PP-3252 Signed-off-by:
-
James Bartlett authored
Summary: Adds a way to keep track of memory use during benchmarks. The MemoryTracker will continually poll the tcmalloc API to determine heap size and physical size of tcmalloc allocated memory. It will keep track of the starting memory, the ending memory, and the max memory between start and end. This will be used in the socket tracer benchmark to determine peak memory usage of different protocol parsers. Test Plan: Currently unused. Future diff will add usage in a benchmark. No need for explicit tests since its only used in benchmarking and not production code. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi, yzhao JIRA Issues: PP-3211 Signed-off-by:
-
Vihang Mehta authored
Summary: Build and use test binaries instead of checking them into the codebase. Test Plan: All existing tests will pass. Reviewers: oazizi, zasgar Reviewed By: oazizi Signed-off-by:
-
Vihang Mehta authored
Summary: The `go get` for the grpc server and client binaries was fetching the entire repo as a dependency to handle the proto import. Instead package up the proto file (and the corresponding gen file since we have it anyway) into a container layer and add it to the images. This removes the need to fetch the repo to get the proto. (Though we still fetch the other deps on demand). Test Plan: All existing tests will pass. Ensured that the generated binaries function as expected and the filesystem in those containers no longer includes a `px.dev/pixie` in the `go/mod` download folder. Reviewers: oazizi, zasgar Reviewed By: oazizi, zasgar Signed-off-by:
-
Vihang Mehta authored
Summary: We are noticing a lot more job timeouts on main after the clang upgrade and some other changes. Try to increase resources and the timeout to address this. Test Plan: Jenkins shouldn't timeout as frequently anymore Reviewers: zasgar, michelle Reviewed By: zasgar Signed-off-by:
-
Hannah Troisi authored
Test Plan: Built & tested the CLI locally. Reviewers: michelle, rcheng Reviewed By: michelle Signed-off-by:
Hannah Troisi <htroisi@pixielabs.ai> Differential Revision: https://phab.corp.pixielabs.ai/D10790 GitOrigin-RevId: fcc2a1a28bf1216b6290c089577d535b9b1fcb64
-
Natalie Serrino authored
Summary: This diff adds the K8s YAML files to deploy the Go profiler load tester to a cluster. When we have other language profilers, we can add a profile to the skaffold file that gets passed in as an input argument. Test Plan: ran it on my cluster Reviewers: zasgar, vihang, oazizi, jamesbartlett Reviewed By: vihang JIRA Issues: PP-3248 Signed-off-by:
-
Zain Asgar authored
Summary: These are used by both client and server so refactoring to move it up higher. Test Plan: Existing Reviewers: michelle, vihang, nserrino Reviewed By: michelle, vihang Signed-off-by:
-
Zain Asgar authored
Summary: No need to nest the main file under the server directory. Also fixes the skaffold file to have the right path. Test Plan: existing Reviewers: vihang Reviewed By: vihang Signed-off-by:
-
Natalie Serrino authored
Summary: This creates a loadtest for the profiler for Golang. It bascically creates N goroutines that each have a random callstack of M functions. This is so that each time we sample, there is a different stack trace (rather than the repetitive stacktrace pattern of demo applications). It is meant to resemble a larger program that may have lots of different stack trace patterns at any time. ``` NUM_GOROUTINES=1 PAUSE_TIME_NS=1000000000 NUM_FUNCTIONS=30 CALL_STACK_DEPTH=2 bazel run :profiler_loadtest_golang ``` would generate a stack trace that looks like this: ``` goroutine 6 [chan receive]: main.(*env).getRandom.func1(0x2) src/e2e_test/profiler_loadtest/go/main.go:43 +0x48 main.(*env).fn29(0xc000016228, 0x1) src/e2e_test/profiler_loadtest/go/main.go:290 +0x50 main.(*env).fn17(0xc000016228, 0x0) src/e2e_test/profiler_loadtest/go/main.go:218 +0x50 main.main.func1(0xc000016228) src/e2e_test/profiler_loadtest/go/main.go:325 +0x46 created by main.main src/e2e_test/profiler_loadtest/go/main.go:322 +0x205 ``` when you access `curl -s http://localhost:6060/debug/pprof/goroutine\?debug\=2` It will always add 12-13 more functions than CALL_STACK_DEPTH: main, the anonymous function of the Nth goroutine, and the crypto functions used to create CPU utilization. Next up: create kubernetes deployment for this so that we can run it on kubernetes. Test Plan: ran it locally and looked at the call stack depth Reviewers: zasgar, oazizi, vihang, jamesbartlett Reviewed By: vihang Subscribers: jps JIRA Issues: PP-3248 Signed-off-by:
-
yzhao1012 authored
Summary: Bundled here is a minor change in conn_id_t's ToString(), where upid is now printed as one unit, from: pid=%d start_time_ticks=%d to upid=%d:%d This reduces the length of CONN_TRACE() logs. Test Plan: Since this just moves code between files, the existing tests already cover them Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Signed-off-by:
-
Hannah Troisi authored
Summary: The demo code lives in this repo: https://github.com/pixie-labs/microservice-kafka/tree/pixie Test Plan: Tested locally by pushing the artifacts to the dev bucket and updating the artifacts flag. Reviewers: michelle Reviewed By: michelle Signed-off-by:
-
James Bartlett authored
Summary: Sets the maximum total size of data perf buffers to be 256MB, and the control buffers to be ~13MB. On a 32 CPU node the total virtual memory usage at startup goes from 2.9GB to ~1GB. Test Plan: Tested on a node with 4cpus, and a node with 32 cpus. Checked that allocated perf_event buffers on a 32 cpu node were about 296MB total (ignoring perf profiler buffers) with this change. compared to >2GB before this change. Checked that allocated perf_event buffers on a 4 cpu node didn't change with this diff, stayed at 280MB. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: yzhao, oazizi Signed-off-by:
-
Hannah Troisi authored
Summary: This yaml file for the Sock Shop demo was creating the wrong namespace. This bug probably went undetected because the CLI manually creates the namespace before deploying the yaml file when running `px demo deploy`. So deploys using the CLI were creating both the `px-sock-shop` and (unnecessary) `sock-shop` namespaces. Test Plan: Built the CLI locally and pass it the dev artifacts. Reviewers: michelle, vihang Reviewed By: vihang Signed-off-by:
-
Hannah Troisi authored
Test Plan: No testing needed, just updating a README.md file. Reviewers: vihang Reviewed By: vihang Signed-off-by:
-
Vihang Mehta authored
Summary: Since the CLI runs on a user's machine, there's a large chance of a clock skew which would cause JWTs to not be valid. We used to not validate these JWTs when parsing with jwt-go, but I added validation when we moved to using jwx. Drop the validation so that CLI functions continue to work on machines with large skew. Test Plan: Pixie CLI should login properly on machines with a large skew. Reviewers: michelle, zasgar Reviewed By: michelle Signed-off-by:
-
- 15 Feb, 2022 5 commits
-
-
James Bartlett authored
Summary: This reverts commit f8dbe50b3c7af48fad9e34574f45fef063fce991. Test Plan: Reverting old commit, tested that reverting fixes perf regression. Reviewers: #stirling, oazizi, yzhao Reviewed By: #stirling, oazizi, yzhao Signed-off-by:
-
Omid Azizi authored
Summary: Remove targets are not the dependencies of any build targets, and are not really used anymore. Test Plan: Rely on Jenkins to ensure build is not broken. Reviewers: #stirling, vihang, yzhao Reviewed By: #stirling, yzhao Signed-off-by:
-
Pete Stevenson authored
Summary: TSIA Test Plan: Tested with demo apps. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi JIRA Issues: PP-2982 Signed-off-by:
-
Pete Stevenson authored
Summary: In testing, we observed the Java process get into a bad state if the attacher tried to attach to soon after the Java process was started. While this may be unlikely in a prod. scenario, we sleep a little bit so that the Java process can get past its 'primordial' phase and into its 'live' phase. Test Plan: Existing. Reviewers: #stirling, oazizi Reviewed By: #stirling, oazizi Subscribers: oazizi JIRA Issues: PP-2982, PP-3237 Signed-off-by:
-
Vihang Mehta authored
Summary: This reduces the compressed size of the base image from ~300MB to ~100MB and speeds up builds and tests. Test Plan: All existing tests should pass. Updated the one failure I saw with `bazel build //...` Reviewers: oazizi, zasgar Reviewed By: oazizi Signed-off-by:
-
