feat: add Postgres support to SearchOperations#3307
feat: add Postgres support to SearchOperations#3307msmithstubbs wants to merge 16 commits intoLogflare:mainfrom
Conversation
msmithstubbs
force-pushed
the
feat/postgres-search
branch
2 times, most recently
from
693cf45 to
ee679c4
Compare
| key = if key == "count", do: "value", else: key | ||
| Map.put(acc, key, normalize_postgres_value(key, value)) |
There was a problem hiding this comment.
needs to factor in custom aggregates. chart would break for custom agg like p99 i think
would also be good to have unit test in logs_search_lv_test.exs |
msmithstubbs
force-pushed
the
feat/postgres-search
branch
4 times, most recently
from
b385b5a to
87671ca
Compare
| |> nested_map_update() | ||
| end | ||
|
|
||
| {:ok, rows} |
There was a problem hiding this comment.
We should make the returned map the same as bigquery so that we don't have to normalise in the caller
There was a problem hiding this comment.
An Adaptor.QueryResult struct might work to make things more standardised
| |> TestUtils.wait_for_render("#logs-list-container li") | ||
|
|
||
| assert view |> element("#logs-list-container") |> render() =~ matching_message | ||
| end |
There was a problem hiding this comment.
Should refute non matching message
| bq_schema = TestUtils.build_bq_schema(%{"event_message" => matching_message}) | ||
|
|
||
| assert {:ok, _source_schema} = | ||
| SourceSchemas.create_or_update_source_schema(source, %{ | ||
| bigquery_schema: bq_schema, | ||
| schema_flat_map: SchemaUtils.bq_schema_to_flat_typemap(bq_schema) | ||
| }) | ||
|
|
There was a problem hiding this comment.
Can be combined into a source_schema factory so that it is a bit more ergonomic.
| schema_flat_map: SchemaUtils.bq_schema_to_flat_typemap(bq_schema) | ||
| }) | ||
|
|
||
| Cachex.clear(Logflare.SourceSchemas.Cache) |
There was a problem hiding this comment.
Don't need to clear cache if it is inserted before source sup is started
There was a problem hiding this comment.
Good point, moved the insert to happen before source sup start.
| end | ||
|
|
||
| test "generates expected SQL for all aggregate types", %{base_so: base_so} do | ||
| for agg <- [:count, :avg, :sum, :max] do |
There was a problem hiding this comment.
We're only testing a subset of the accepted aggs?
There was a problem hiding this comment.
Had thought of this more as an integration test that aggregate types work since each agg is covered by postgres backend transformer test. But it's cheap test to do here, too.
Added full list: :count, :avg, :sum, :max, :countd, :p50, :p95, :p99
msmithstubbs
force-pushed
the
feat/postgres-search
branch
from
44d6724 to
a955253
Compare
msmithstubbs
force-pushed
the
feat/postgres-search
branch
from
a955253 to
cbd6465
Compare
2 participants
Adds support for searches against a postgres backend.
backend_typefield to%SearchOperation{}, defaults to `:bigquery', and is set automatically to default backend.SearchOperationsbuilds the search query for the appropriate backendIntended for integration tests in CI. Part of #3239
Follows on from #3306 and #3317
test_connection/1in test setup