This project is mirrored from https://gitee.com/mirrors/Mizu.git. Pull mirroring failed .
Repository mirroring has been paused due to too many failed attempts. It can be resumed by a project maintainer.
  1. 08 May, 2022 1 commit
  2. 02 May, 2022 1 commit
    • M. Mert Yıldıran's avatar
      Add GraphQL detection (#980) · 3cbccccb
      M. Mert Yıldıran authored
      * Add GraphQL detection
      
      * Change the abbreviation to `GQL`
      
      * Fix the `TestMacros` test case
      
      * Try to fix the security issues reported by Snyk
      
      * Run `go mod tidy` in `agent` directory
      
      * Upgrade `golang.org/x/crypto`
      
      * Downgrade `golang.org/x/crypto`
      
      * Downgrade `golang.org/x/crypto`
      
      * Downgrade `golang.org/x/crypto`
      
      * Upgrade to `github.com/vektah/gqlparser/v2 v2.4.2`
      
      * Run `go mod tidy`
      
      * Replace `github.com/vektah/gqlparser/v2` with its fork `github.com/mertyildiran/gqlparser/v2` to fix the vulns
      
      * Upgrade the `github.com/mertyildiran/gqlparser/v2 v2.4.6`
      
      * Don't upgrade `golang.org/x/net`
      Unverified
      3cbccccb
  3. 28 Apr, 2022 1 commit
    • M. Mert Yıldıran's avatar
      Refactor `tap` module to achieve synchronously closing other protocol... · d3e6a69d
      M. Mert Yıldıran authored
      Refactor `tap` module to achieve synchronously closing other protocol dissectors upon identification (#1026)
      
      * Remove `tcpStreamWrapper` struct
      
      * Refactor `tap` module and move some of the code to `tap/api` module
      
      * Move `TrafficFilteringOptions` struct to `shared` module
      
      * Change the `Dissect` method signature to have `*TcpReader` as an argument
      
      * Add `CloseOtherProtocolDissectors` method and use it to synchronously close the other protocol dissectors
      
      * Run `go mod tidy` in `cli` module
      
      * Rename `SuperIdentifier` struct to `ProtoIdentifier`
      
      * Remove `SuperTimer` struct
      
      * Bring back `CloseTimedoutTcpStreamChannels` method
      
      * Run `go mod tidy` everywhere
      
      * Remove `GOGC` environment variable from tapper
      
      * Fix the tests
      
      * Bring back `debug.FreeOSMemory()` call
      
      * Make `CloseOtherProtocolDissectors` method mutexed
      
      * Revert "Remove `GOGC` environment variable from tapper"
      
      This reverts commit cfc2484b.
      
      * Bring back the removed `checksum`, `nooptcheck` and `ignorefsmerr` flags
      
      * Define a bunch of interfaces and don't export any new structs from `tap/api`
      
      * Keep the interfaces in `tap/api` but move the structs to `tap/tcp`
      
      * Fix the unit tests by depending on `github.com/up9inc/mizu/tap`
      
      * Use the modified `tlsEmitter`
      
      * Define `TlsChunk` interface and make `tlsReader` implement `TcpReader`
      
      * Remove unused fields in `tlsReader`
      
      * Define `ReassemblyStream` interface and separate `gopacket` specififc fields to `tcpReassemblyStream` struct
      
      Such that make `tap/api` don't depend on `gopacket`
      
      * Remove the unused fields
      
      * Make `tlsPoller` implement `TcpStream` interface and remove the call to `NewTcpStreamDummy` method
      
      * Remove unused fields from `tlsPoller`
      
      * Remove almost all of the setter methods in `TcpReader` and `TcpStream` interface and remove `TlsChunk` interface
      
      * Revert "Revert "Remove `GOGC` environment variable from tapper""
      
      This reverts commit ab2b9a80.
      
      * Revert "Bring back `debug.FreeOSMemory()` call"
      
      This reverts commit 1cce863b.
      
      * Remove excess comment
      
      * Fix acceptance tests (`logger` module) #run_acceptance_tests
      
      * Bring back `github.com/patrickmn/go-cache`
      
      * Fix `NewTcpStream` method signature
      
      * Put `tcpReader` and `tcpStream` mocks into protocol dissectors to remove `github.com/up9inc/mizu/tap` dependency
      
      * Fix AMQP tests
      
      * Revert 960ba644
      
      * Revert `go.mod` and `go.sum` files in protocol dissectors
      
      * Fix the comment position
      
      * Revert `AppStatsInst` change
      
      * Fix indent
      
      * Fix CLI build
      
      * Fix linter error
      
      * Fix error msg
      
      * Revert some of the changes in `chunk.go`
      Unverified
      d3e6a69d
  4. 06 Apr, 2022 1 commit
  5. 21 Mar, 2022 1 commit
  6. 17 Mar, 2022 1 commit
  7. 06 Mar, 2022 1 commit
    • M. Mert Yıldıran's avatar
      Always derive the summary and method fields from the entry in the database on read (#877) · dd430c31
      M. Mert Yıldıran authored
      * Always derive the summary and method fields from the entry in the database on read
      
      * Update the expected JSONs in the protocol unit tests
      
      * Add test cases for `Summarize` method
      
      * Remove unused `GetEntry` method, `DataUnmarshaler` struct and `UnmarshalData` method
      
      * Temporarily enable the acceptance tests
      
      * Temporarily disable Slack notification on failure
      
      * Update the Cypress tests
      
      * Fix an issue in Redis
      
      * Fix a typo and the Cypress tests
      
      * Revert "Temporarily disable Slack notification on failure"
      
      This reverts commit cad1901e.
      
      * Revert "Temporarily enable the acceptance tests"
      
      This reverts commit bad7706c.
      Unverified
      dd430c31
  8. 21 Feb, 2022 1 commit
  9. 15 Feb, 2022 3 commits
  10. 14 Feb, 2022 1 commit
  11. 01 Feb, 2022 1 commit
  12. 30 Jan, 2022 1 commit
  13. 25 Jan, 2022 1 commit
    • M. Mert Yıldıran's avatar
      Add ARM64 and cross-compilation support to the agent image (#659) · a9a61edd
      M. Mert Yıldıran authored
      * modified Dockerfile to work for both amd64 (Intel) and arm64 (M1)
      
      * added changelog
      
      * Update `Dockerfile` to have `ARCH` build argument
      
      * Remove `docs/CHANGES.md`
      
      * Upgrade the Basenine version from `v0.3.0` to `v0.4.6`
      
      * Update `publish.yml` to have `ARCH` build argument
      
      * Switch `BasenineImageRepo` to Docker Hub
      
      * Have separate build arguments for `ARCH` and `GOARCH`
      
      * Upgrade the Basenine version from `v0.4.6` to `v0.4.10`
      
      * Oops forgot to update the 10th duplicated shell script
      
      * Fix the oopsie and reduce duplications
      
      * Fix `Dockerfile`
      
      * Fix the incompatibility issue between Go plugins and gold linker in Alpine inside `Dockerfile`
      
      * Fix `asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access` error
      
      * Update `Dockerfile` to have cross-compilation on an AMD64 machine
      
      Also revert changes in the shell scripts
      
      * Delete `debug.Dockerfile`
      
      * Create a cus...
      Unverified
      a9a61edd
  14. 22 Jan, 2022 1 commit
  15. 20 Jan, 2022 1 commit
  16. 16 Jan, 2022 1 commit
  17. 11 Jan, 2022 1 commit
    • M. Mert Yıldıran's avatar
      Upgrade Basenine to `0.3.0`, do a refactor to enable `redact` helper and... · 80e881fe
      M. Mert Yıldıran authored
      Upgrade Basenine to `0.3.0`, do a refactor to enable `redact` helper and update the cheatsheet (#614)
      
      * Upgrade Basenine version from `0.2.26` to `0.3.0`
      
      * Remove `Summarize` method from `Dissector` interface and refactor the data structures in `tap/api/api.go`
      
      * Rename `MizuEntry` to `Entry` and `BaseEntryDetails` to `BaseEntry`
      
      * Populate `ContractStatus` field as well
      
      * Update the cheatsheet
      
      * Upgrade the Basenine version in the helm chart as well
      
      * Remove a forgoten `console.log` call
      Unverified
      80e881fe
  18. 19 Dec, 2021 1 commit
    • Igor Gov's avatar
      Develop -> main (#544) · 72f47536
      Igor Gov authored
      
      * Add support of listening to multiple netns (#418)
      
      * multiple netns listen - initial commit
      
      * multiple netns listen - actual work
      
      * remove redundant log line
      
      * map /proc of host to tapper
      
      * changing kubernetes provider again after big conflict
      
      * revert node-sass version back to 5.0.0
      
      * Rename host_source to hostSource
      Co-authored-by: default avatargadotroee <55343099+gadotroee@users.noreply.github.com>
      
      * PR fixes - adding comment + typos + naming conventions
      
      * go fmt + making procfs read only
      
      * setns back to the original value after packet source initialized
      Co-authored-by: default avatargadotroee <55343099+gadotroee@users.noreply.github.com>
      
      * TRA-3842 daemon acceptance tests (#429)
      
      * Update tap_test.go and testsUtils.go
      
      * Update tap_test.go
      
      * Update testsUtils.go
      
      * Update tap_test.go and testsUtils.go
      
      * Update tap_test.go and testsUtils.go
      
      * Update testsUtils.go
      
      * Update tap_test.go
      
      * gofmt
      
      * TRA-3913 support mizu via expose service (#440)
      ...
      Unverified
      72f47536
  19. 14 Dec, 2021 1 commit
  20. 25 Nov, 2021 2 commits
  21. 22 Nov, 2021 2 commits
    • M. Mert Yıldıran's avatar
    • M. Mert Yıldıran's avatar
      Make the `gRPC` and `HTTP/2` distinction (#492) · 12ca3d87
      M. Mert Yıldıran authored
      * Remove the extra negation on `nodefrag` flag's value
      
      * Support IPv4 fragmentation and IPv6 at the same time
      
      * Set `Method` and `StatusCode` fields correctly for `HTTP/2`
      
      * Replace unnecessary `grpc` naming with `http2`
      
      * Make the `gRPC` and `HTTP/2` distinction
      
      * Fix the macros of `http` extension
      
      * Fix the macros of other protocol extensions
      
      * Update the method signature of `Represent`
      
      * Fix the `HTTP/2` support
      
      * Fix some minor issues
      
      * Upgrade Basenine version from `0.2.10` to `0.2.11`
      
      Sorts macros before expanding them and prioritize the long macros.
      
      * Don't regex split the gRPC method name
      
      * Re-enable `nodefrag` flag
      Unverified
      12ca3d87
  22. 16 Nov, 2021 1 commit
  23. 15 Nov, 2021 3 commits
    • M. Mert Yıldıran's avatar
      Prevent `elapsedTime` to be negative (#467) · a6bf39fa
      M. Mert Yıldıran authored
      Also fix the `elapsedTime` for Redis.
      Unverified
      a6bf39fa
    • M. Mert Yıldıran's avatar
    • M. Mert Yıldıran's avatar
      Optimize UI entry feed performance (#452) · 618cb3a4
      M. Mert Yıldıran authored
      * Optimize the React code for feeding the entries
      
      By building `EntryItem` only once and updating the `entries` state on meta query messages.
      
      * Upgrade `react-scrollable-feed-virtualized` version from `1.4.3` to `1.4.8`
      
      * Fix the `isSelected` state
      
      * Set the query text before deciding the background to prevent lags while typing
      
      * Upgrade Basenine version from `0.2.6` to `0.2.7`
      
      * Set the query background color only if the query is same after the HTTP request and use `useEffect` instead
      
      * Upgrade Basenine version from `0.2.7` to `0.2.8`
      
      * Use `CancelToken` of `axios` instead of trying to check the query state
      
      * Turn `updateQuery` function into a state hook
      
      * Update the macro for `http`
      
      * Do the `source.cancel()` call in `axios.CancelToken`
      
      * Reduce client-side logging
      Unverified
      618cb3a4
  24. 09 Nov, 2021 1 commit
    • M. Mert Yıldıran's avatar
      Migrate from SQLite to Basenine and introduce a new filtering syntax (#279) · d2fe3f66
      M. Mert Yıldıran authored
      * Fix the OOMKilled error by calling `debug.FreeOSMemory` periodically
      
      * Remove `MAX_NUMBER_OF_GOROUTINES` environment variable
      
      * Change the line
      
      * Increase the default value of `TCP_STREAM_CHANNEL_TIMEOUT_MS` to `10000`
      
      * Write the client and integrate to the new real-time database
      
      * Refactor the WebSocket implementaiton for `/ws`
      
      * Adapt the UI to the new filtering system
      
      * Fix the rest of the issues in the UI
      
      * Increase the buffer of the scanner
      
      * Implement accessing single records
      
      * Increase the buffer of another scanner
      
      * Populate `Request` and `Response` fields of `MizuEntry`
      
      * Add syntax highlighting for the query
      
      * Add database to `Dockerfile`
      
      * Fix some issues
      
      * Update the `realtime_dbms` Git module commit hash
      
      * Upgrade Gin version and print the query string
      
      * Revert "Upgrade Gin version and print the query string"
      
      This reverts commit aa09f904...
      Unverified
      d2fe3f66
  25. 19 Oct, 2021 1 commit
  26. 23 Sep, 2021 1 commit
  27. 18 Sep, 2021 1 commit
  28. 13 Sep, 2021 1 commit
  29. 12 Sep, 2021 1 commit
    • M. Mert Yıldıran's avatar
      Stop the hanging Goroutines by dropping the old, unidentified TCP streams (#260) · 858a6468
      M. Mert Yıldıran authored
      * Close the hanging TCP message channels after a dynamically aligned timeout (base `10000` milliseconds)
      
      * Bring back `source.Lazy`
      
      * Add a one more `sync.Map.Delete` call
      
      * Improve the formula by taking base Goroutine count into account
      
      * Reduce duplication
      
      * Include the dropped TCP streams count into the stats tracker and print a debug log whenever it happens
      
      * Add `superIdentifier` field to `tcpStream` to check if it has identified
      
      Also stop the other protocol dissectors if a TCP stream identified by a protocol.
      
      * Take one step forward in fixing the channel closing issue (WIP)
      
      Add `sync.Mutex` to `tcpReader` and make the loops reference based.
      
      * Fix the channel closing issue
      
      * Improve the accuracy of the formula, log better and multiply `baseStreamChannelTimeoutMs` by 100
      
      * Remove `fmt.Printf`
      
      * Replace `runtime.Gosched()` with `time.Sleep(1 * time.Millisecond)`
      
      * Close the channels of other protocols in case of an identification
      
      * Simplify the logic
      
      * Replace the formula with hard timeout 5000 milliseconds and 4000 maximum number of Goroutines
      Unverified
      858a6468
  30. 07 Sep, 2021 1 commit
  31. 05 Sep, 2021 1 commit
  32. 04 Sep, 2021 1 commit
  33. 02 Sep, 2021 1 commit
    • M. Mert Yıldıran's avatar
      Refactor Mizu, define an extension API and add new protocols: AMQP, Kafka (#224) · 366c1d0c
      M. Mert Yıldıran authored
      * Separate HTTP related code into `extensions/http` as a Go plugin
      
      * Move `extensions` folder into `tap` folder
      
      * Move HTTP files into `tap/extensions/lib` for now
      
      * Replace `orcaman/concurrent-map` with `sync.Map`
      
      * Remove `grpc_assembler.go`
      
      * Remove `github.com/up9inc/mizu/tap/extensions/http/lib`
      
      * Add a build script to automatically build extensions from a known path and load them
      
      * Start to define the extension API
      
      * Implement the `run()` function for the TCP stream
      
      * Add support of defining multiple ports to the extension API
      
      * Set the extension name inside the extension
      
      * Declare the `Dissect` function in the extension API
      
      * Dissect HTTP request from inside the HTTP extension
      
      * Make the distinction of outbound and inbound ports
      
      * Dissect HTTP response from inside the HTTP extension
      
      * Bring back the HTTP request-response pair matcher
      
      * Return a `*api.RequestResponsePair` from the dissection
      
      * Bring back the gRPC-HTTP/2 parser
      
      * Fix the issues in `handleHTTP1ClientStream` and `handleHTTP1ServerStream`
      
      * Call a function pointer to emit dissected data back to the `tap` package
      
      * roee changes -
      trying to fix agent to work with the "api" object) - ***still not working***
      
      * small mistake in the conflicts
      
      * Fix the issues that are introduced by the merge conflict
      
      * Add `Emitter` interface to the API and send `OutputChannelItem`(s) to `OutputChannel`
      
      * Fix the `HTTP1` handlers
      
      * Set `ConnectionInfo` in HTTP handlers
      
      * Fix the `Dockerfile` to build the extensions
      
      * remove some unwanted code
      
      * no message
      
      * Re-enable `getStreamProps` function
      
      * Migrate back from `gopacket/tcpassembly` to `gopacket/reassembly`
      
      * Introduce `HTTPPayload` struct and `HTTPPayloader` interface to `MarshalJSON()` all the data structures that are returned by the HTTP protocol
      
      * Read `socketHarOutChannel` instead of `filteredHarChannel`
      
      * Connect `OutputChannelItem` to the last WebSocket means that finally the web UI started to work again
      
      * Add `.env.example` to React app
      
      * Marshal and unmarshal `*http.Request`, `*http.Response` pairs
      
      * Move `loadExtensions` into `main.go` and map extensions into `extensionsMap`
      
      * Add `Summarize()` method to the `Dissector` interface
      
      * Add `Analyze` method to the `Dissector` interface and `MizuEntry` to the extension API
      
      * Add `Protocol` struct and make it effect the UI
      
      * Refactor `BaseEntryDetails` struct and display the source and destination ports in the UI
      
      * Display the protocol name inside the details layout
      
      * Add `Represent` method to the `Dissector` interface and manipulate the UI through this method
      
      * Make the protocol color affect the details layout color and write protocol abbreviation vertically
      
      * Remove everything HTTP related from the `tap` package and make the extension system fully functional
      
      * Fix the TypeScript warnings
      
      * Bring in the files related AMQP into `amqp` directory
      
      * Add `--nodefrag` flag to the tapper and bring in the main AMQP code
      
      * Implement the AMQP `BasicPublish` and fix some issues in the UI when the response payload is missing
      
      * Implement `representBasicPublish` method
      
      * Fix several minor issues
      
      * Implement the AMQP `BasicDeliver`
      
      * Implement the AMQP `QueueDeclare`
      
      * Implement the AMQP `ExchangeDeclare`
      
      * Implement the AMQP `ConnectionStart`
      
      * Implement the AMQP `ConnectionClose`
      
      * Implement the AMQP `QueueBind`
      
      * Implement the AMQP `BasicConsume`
      
      * Fix an issue in `ConnectionStart`
      
      * Fix a linter error
      
      * Bring in the files related Kafka into `kafka` directory
      
      * Fix the build errors in Kafka Go files
      
      * Implement `Dissect` method of Kafka and adapt request-response pair matcher to asynchronous client-server stream
      
      * Do the "Is reversed?" checked inside `getStreamProps` and fix an issue in Kafka `Dissect` method
      
      * Implement `Analyze`, `Summarize` methods of Kafka
      
      * Implement the representations for Kafka `Metadata`, `RequestHeader` and `ResponseHeader`
      
      * Refactor the AMQP and Kafka implementations to create the summary string only inside the `Analyze` method
      
      * Implement the representations for Kafka `ApiVersions`
      
      * Implement the representations for Kafka `Produce`
      
      * Implement the representations for Kafka `Fetch`
      
      * Implement the representations for Kafka `ListOffsets`, `CreateTopics` and `DeleteTopics`
      
      * Fix the encoding of AMQP `BasicPublish` and `BasicDeliver` body
      
      * Remove the unnecessary logging
      
      * Remove more logging
      
      * Introduce `Version` field to `Protocol` struct for dynamically switching the HTTP protocol to HTTP/2
      
      * Fix the issues in analysis and representation of HTTP/2 (gRPC) protocol
      
      * Fix the issues in summary section of details layout for HTTP/2 (gRPC) protocol
      
      * Fix the read errors that freezes the sniffer in HTTP and Kafka
      
      * Fix the issues in HTTP POST data
      
      * Fix one more issue in HTTP POST data
      
      * Fix an infinite loop in Kafka
      
      * Fix another freezing issue in Kafka
      
      * Revert "UI Infra - Support multiple entry types + refactoring (#211)"
      
      This reverts commit f74a52d4.
      
      * Fix more issues that are introduced by the merge
      
      * Fix the status code in the summary section
      
      * adding the cleaner again (why we removed it?).
      add TODO: on the extension loop .
      
      * fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists
      
      * fix GetEntrties ("/entries" endpoint) - working with "tapApi.BaseEntryDetail" (moved from shared)
      
      * Fix an issue in the UI summary section
      
      * Refactor the protocol payload structs
      
      * Fix a log message in the passive tapper
      
      * Adapt `APP_PORTS` environment variable to the new extension system and change its format to `APP_PORTS='{"http": ["8001"]}' `
      
      * Revert "fix dockerfile (remove deleting .env file) - it is found in dockerignore and fails to build if the file not exists"
      
      This reverts commit 4f514ae1.
      
      * Bring in the necessary changes from f74a52d4
      
      * Open the API server URL in the web browser as soon as Mizu is ready
      
      * Make the TCP reader consists of a single Go routine (instead of two) and try to dissect in both client and server mode by rewinding
      
      * Swap `TcpID` without overwriting it
      
      * Sort extension by priority
      
      * Try to dissect with looping through all the extensions
      
      * fix getStreamProps function.
      (it should be passed from CLI as it was before).
      
      * Turn TCP reader back into two Goroutines (client and server)
      
      * typo
      
      * Learn `isClient` from the TCP stream
      
      * Set `viewer` style `overflow: "auto"`
      
      * Fix the memory leaks in AMQP and Kafka dissectors
      
      * Revert some of the changes in be7c65eb
      
      * Remove `allExtensionPorts` since it's no longer needed
      
      * Remove `APP_PORTS` since it's no longer needed
      
      * Fix all of the minor issues in the React code
      
      * Check Kafka header size and fail-fast
      
      * Break the dissectors loop upon a successful dissection
      
      * Don't break the dissector loop. Protocols might collide
      
      * Improve the HTTP request-response counter (still not perfect)
      
      * Make the HTTP request-response counter perfect
      
      * Revert "Revert some of the changes in be7c65eb"
      
      This reverts commit 08e7d786.
      
      * Bring back `filterItems` and `isHealthCheckByUserAgent` functions
      
      * Remove some development artifacts
      
      * remove unused and commented lines that are not relevant
      
      * Fix the performance in TCP stream factory. Make it create two `tcpReader`(s) per extension
      
      * Change a log to debug
      
      * Make `*api.CounterPair` a field of `tcpReader`
      
      * Set `isTapTarget` to always `true` again since `filterAuthorities` implementation has problems
      
      * Remove a variable that's only used for logging even though not introduced by this branch
      
      * Bring back the `NumberOfRules` field of `ApplicableRules` struct
      
      * Remove the unused `NewEntry` function
      
      * Move `k8sResolver == nil` check to a more appropriate place
      
      * default healthChecksUserAgentHeaders should be empty array (like the default config value)
      
      * remove spam console.log
      
      * Rules button cause app to crash (access the service via incorrect property)
      
      * Ignore all .env* files in docker build.
      
      * Better caching in dockerfile: only copy go.mod before go mod download.
      
      * Check for errors while loading an extension
      
      * Add a comment about why `Protocol` is not a pointer
      
      * Bring back the call to `deleteOlderThan`
      
      * Remove the `nil` check
      
      * Reduce the maximum allowed AMQP message from 128MB to 1MB
      
      * Fix an error that only occurs when a Kafka broker is initiating
      
      * Revert the change in b2abd7b9
      
      
      
      * Fix the service name resolution in all protocols
      
      * Remove the `anydirection` flag and fix the issue in `filterAuthorities`
      
      * Pass `sync.Map` by reference to `deleteOlderThan` method
      
      * Fix the packet capture issue in standalone mode that's introduced by the removal of `anydirection`
      
      * Temporarily resolve the memory exhaustion in AMQP
      
      * Fix a nil pointer dereference error
      
      * Fix the CLI build error
      
      * Fix a memory leak that's identified by `pprof`
      Co-authored-by: default avatarRoee Gadot <roee.gadot@up9.com>
      Co-authored-by: default avatarNimrod Gilboa Markevich <nimrod@up9.com>
      Unverified
      366c1d0c