R*aj3dZddlmZddlZddlZddlZddlmZmZddl m Z m Z ddl m Z ddl mZmZmZmZmZmZe dZdd Zdd Zdd Zdd Zdd Zedk(reyy)zGRead-only deterministic analysis and clustering for the review backlog.) annotationsN)Counter defaultdict)datetimetimezone)Path)NORMALIZATION_VERSIONVERSIONalias_registryclassify exact_keynormz?/opt/struktur/obsidian-knowledge-pipeline/knowledge_pipeline.dbcdtjtjj S)N)rnowrutc isoformat:/opt/struktur/knowledge-curator/minimal_review_analysis.pyrrs << % / / 11rc"|dkrdS|dkrdSdS)N(shortmediumlongr)lengths rquality_bucketrsrk7K6CSELECT candidate_json FROM truth_items WHERE status='approved') sqlite3connectPIPEsetexecuteaddr jsonloads Exceptionclose)coutpayloads r approved_keysr0s%vX.D9A %Cii `a   GGIdjj12 3 GGI J   s-B B Bc#Pt}tjdtdd}tj|_t }g}|jdD]R}tj|d}t||}|j|d|d|d |j|T|jtt}tt} tt} |D]}||d j|d j!|d |d |dt#|dt%|df} | |d<| | j|| |dj|g} d} t'| j)dD]\}}t+|dkr| dz } d| d}|D]}||d<t+||d<| j||t+|t+|Dchc]}|d c}t'|Dchc]}|d  c}t'|Dchc]}|d c}t'|Dchc]}|d c}dd|ddDcgc]}|d c}t'|Dchc]}t#|d c}dd!d"d# |D]:}|j-dd|j-ddt+||d |d$<<| Dcgc]}t/d%|d&Ds|}}| Dcgc]}t/d'|d&Dr|}}|D]}d(|d< t1d)|D}g}| j)D]\}}|Dchc]}t#|d*}}t+|dkDs0t+|dkDs?|j|t+|t'|t+|Dchc]}t#|d c}|ddDcgc]}|d c}d+d,t3d-|D}t3d.|D}t3d/|D}t3d0| D}t1d1| D}t3d2|D}t3d3|D} t3d4|D}!id5|d6t4d7t6d8t9td9t+|d&t;|d:t;|d;t;|d|d?t+|d@|dAt+|dBt+||z dCt+|dDt+||z |t+|z t+|t1dE|D|j=dFdt?| jAdD"#cgc] \}"}#|"|#dG c}#}"|jAdD"#cgc] \}"}#|"|#dH c}#}"|!jAdD"#cgc] \}"}#|"|#dI c}#}"tCd|j=dJdtCd|tCd|j=dKdtCd|j=dLdtCdt1dM|DtCdN|j=dFddOdd| ||dP}$tE|jjGtjH|$dQdtRdSzdTUtj|jJ}%|%jMdV|%jdWDchc]}|d }&}dX|&vr|%jdYdZtOjtPjRjUd[z}'t}(|%jd\|'t4t6||(t+|t+| t+|d]f |%jWd^|$d_D)cgc]})|)d`|)da|)db|)dc|)dd|)dfc})|%jde|%jdf|%jdg|%jdh|%jde|%jWdi|D*cgc]}*|*d|'|*d|*d|*dj|*dk|*dltjH|*d dQm|*dn|*d*|*do|*dp|*d|*d |*dq|*d|*d |*d |*d|*d|*d|*dr|*dtY|*dst9|*dj=dtxsij=dXxs|*j=d xsdufc}*|%jWdv| Dcgc]K}|d|'|d|dw|dxtjH|d&dQmtjH|d;dQm|dfMc}|%jWdy| D*cgc]+}|D]$}*|*j=d|dk(s|d|*df&-c}*}|%jWdz|Dcgc].}|d|'|dwtjH|d{dQm|d||df0c}|%j[|%jd}dud~|dt+|dduddug}+|+t'|j)dD"#cgc]\}"}#d|"d|#dc}#}"z }+|+duddt+| ddt+|dd|ddt+||z ddt+|dduddug z }+|+|$dj)D"#cgc]\}"}#d|"d|#dc}#}"z }+|+gdz }+|+t]|$ddD,cgc]\},}d|,d|dd|ddddc}},z }+tE|j^jGdSj!|+dSzdTUdD"cic]}"|"|$|" c}"Scc}wcc}wcc}wcc}wcc}wcc}wcc}wcc}wcc}wcc}wcc}wcc}#}"wcc}#}"wcc}#}"wcc}wcc})wcc}*wcc}wcc}*}wcc}wcc}#}"wcc}#}"wcc}},wcc}"w)Nrr Tr!zSELECT candidate_id,path,chunk_key,candidate_json,status FROM review_items WHERE status='pending_review' ORDER BY created_at,candidate_idcandidate_json chunk_keystatus)r3pipeline_statusr2r |structural_keycategorysensitivity_classextraction_ruleevidence_length cluster_key conflict_keyrc*t|d |dfSNr)lenpairs rzanalyze..>sCQL=RVWXRYBZr)keyr@z mhr-cluster-05d cluster_id cluster_sizepathd candidate_idobjectgrouped_pending_review) rHr<candidate_countdocument_count categoriessensitivity_classespathsrepresentative_candidate_idsvariable_objectsr4exact_group_sizec3$K|]}|dv yw>invalid_fragmentexcluded_non_knowledgeNr.0r8s r zanalyze..Ls!5UyAXEs5s5UrSc3$K|]}|dv ywrZrr]s rr_zanalyze..Ms!=]AIXM{={=]r`automatic_nonknowledge_clusterc3&K|] }|d ywrQNrr^xs rr_zanalyze..PsE+,Epolarityconflict_pending_review)r=rQ polaritiesobject_variants candidate_idsr4c3&K|] }|d yw)r8Nrr^is rr_zanalyze..Xs61:6rgc3&K|] }|d yw)candidate_typeNrrns rr_zanalyze..Ys=!!,-=rgc3&K|] }|d yw)r9Nrrns rr_zanalyze..Zs GA#6!7 Grgc3&K|] }|d ywrdrr^rows rr_zanalyze..[sKsC 12Krgc3&K|] }|d ywrdrrts rr_zanalyze..\sAS#'(Argc3&K|] }|d yw)rJNrrns rr_zanalyze..]s/Aai/rgc3&K|] }|d yw)r<Nrrns rr_zanalyze..^s7Aq'7rgc3&K|] }|d yw)r:Nrrns rr_zanalyze.._s8QA'(8rg generated_atanalysis_versionnormalization_versiondatabase pending_totalcandidate_typesrT cluster_countcluster_size_distributionclustered_candidateshuman_cluster_counthuman_clustered_candidatesautomatic_cluster_countunclustered_candidatesestimated_human_group_decisions*estimated_unavoidable_individual_decisionsc3&K|] }|d ywrdrres rr_zanalyze..hsP}jkQRSdQeP}rgr[)patterncount)rJr)ruler"inherited_human_decision_candidateindividual_pending_reviewsensitive_pending_reviewc3&K|] }|d ywrdrres rr_zanalyze..ms0aN{|bcdubvaNrg2)*automatically_safe_or_inherited_candidatescluster_assignmentsindividual_unclear sensitive conflictsparser_corrections)+estimated_reduction_in_individual_decisionsconflict_cluster_countconflict_candidatesincomplete_evidencealiasestop_100_patternstop_100_sourcestop_100_extraction_rules sample_sizesno_status_changesno_neo4j_writesclustersritemsF ensure_asciiindentdefault zutf-8)encodinga CREATE TABLE IF NOT EXISTS analysis_runs(run_id TEXT PRIMARY KEY, analysis_version TEXT, normalization_version TEXT, started_at TEXT, completed_at TEXT, candidate_count INTEGER, cluster_count INTEGER, conflict_count INTEGER, status TEXT); CREATE TABLE IF NOT EXISTS candidate_analysis(candidate_id TEXT PRIMARY KEY, run_id TEXT, path TEXT, chunk_key TEXT, candidate_type TEXT, subject TEXT, predicate TEXT, object_json TEXT, modality TEXT, polarity TEXT, temporal_scope TEXT, authority_scope TEXT, sensitivity_class TEXT, category TEXT, classification_reason TEXT, extraction_rule TEXT, exact_key TEXT, structural_key TEXT, conflict_key TEXT, cluster_id TEXT, cluster_size INTEGER, evidence_hash TEXT, evidence_length INTEGER, provenance_complete INTEGER); CREATE INDEX IF NOT EXISTS candidate_analysis_category_idx ON candidate_analysis(category); CREATE INDEX IF NOT EXISTS candidate_analysis_cluster_idx ON candidate_analysis(cluster_id); CREATE TABLE IF NOT EXISTS clusters(cluster_id TEXT PRIMARY KEY, run_id TEXT, cluster_key TEXT, candidate_count INTEGER, document_count INTEGER, categories_json TEXT, sensitivity_json TEXT, status TEXT); CREATE TABLE IF NOT EXISTS cluster_members(cluster_id TEXT, candidate_id TEXT, PRIMARY KEY(cluster_id,candidate_id)); CREATE TABLE IF NOT EXISTS conflict_clusters(conflict_key TEXT PRIMARY KEY, run_id TEXT, candidate_count INTEGER, polarities_json TEXT, object_variants INTEGER, status TEXT); CREATE TABLE IF NOT EXISTS aliases(kind TEXT, alias TEXT, canonical TEXT, rationale TEXT, version TEXT, status TEXT, PRIMARY KEY(kind,alias)); CREATE TABLE IF NOT EXISTS audit(audit_id TEXT PRIMARY KEY, run_id TEXT, phase TEXT, entity_key TEXT, details_json TEXT, created_at TEXT); z%PRAGMA table_info(candidate_analysis) evidence_textzINSERT OR REPLACE INTO analysis_runs VALUES(?,?,?,?,?,?,?,?,?)dry_run_completez2INSERT OR REPLACE INTO aliases VALUES(?,?,?,?,?,?)rkindalias canonical rationaleversionzDELETE FROM candidate_analysiszDELETE FROM clusterszDELETE FROM cluster_memberszDELETE FROM conflict_clustersaINSERT INTO candidate_analysis(candidate_id,run_id,path,chunk_key,candidate_type,subject,predicate,object_json,modality,polarity,temporal_scope,authority_scope,sensitivity_class,category,classification_reason,extraction_rule,exact_key,structural_key,conflict_key,cluster_id,cluster_size,evidence_hash,evidence_length,provenance_complete,evidence_text) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)rqsubject predicate)rmodalitytemporal_scopeauthority_scopeclassification_reason evidence_hashprovenance_complete provenancez,INSERT INTO clusters VALUES(?,?,?,?,?,?,?,?)rQrRz'INSERT INTO cluster_members VALUES(?,?)z1INSERT INTO conflict_clusters VALUES(?,?,?,?,?,?)rjrku"# Minimal Human Review – Dry-Runz Erzeugt: zKandidaten: **z**z ## Kategorienc|d |dfSr?rrBs rrDzanalyze..sZ^_`ZaYabfghbiXjrz- `z`: **z ## Clusterz- Strukturcluster gesamt: **z - Menschliche Gruppencluster: **z)- Kandidaten in menschlichen Clustern: **z- Einzelkandidaten: **z- Konfliktcluster: **z## Stichprobenr)rz## SicherheitsgrenzerumSchreibfreier Dry-Run: keine Statuswerte, Carlo-Entscheidungen, Obsidian- oder Neo4j-Daten wurden verändert.rz## Top-100-Musterrrz- z. ru Kandidaten – `r`)rzr~rSrrrrrrrrrrr)0rr#r$r%Row row_factoryr0r'r)r*r updateappendr,rlistjoinrrsortedrrA setdefaultanysumrr r strdictgetr most_commonminr write_textdumpssqlite executescriptrrrstrftime executemanyintcommit enumeratemarkdown)-argsstartedr-approvedrru candidateitemexactrrr< cluster_rowscluster_counterrEmembersrHmembermrfhuman_clustersautomatic_clusterscluster human_grouped conflict_rowsrjrS type_countssensitivity_counts cluster_sizesgroupedsourcespatternsruleskvr.dbexisting_columnsrun_id completedarolinesns- ranalyzer%seG%vX.D9AKKAMH EyyefJJs#345  8, #k"2s8}hqrs T  GGI  E4 HD!I5 d; ''-hh%5 6Z8H$ObJceijnpAkBfCESTXYjTkEl mn )]$$T*$~&'..t4 5LOx~~/5Z[N W w " 2 :c^abi^jBW^C_RSDEFLDMC_`pvOVwWJKxyzDxEwWpXqwY`xaTUyz{NyOxaqbmsHOtPCDuvw}u~tPmQRVSVmWU\]_^_U`yaPQz{|JzKyaw}Za~bUVCDEFNDOP~bwcdgegwhtLM NNA  d+ *#&uT+->'?#@  A ".VA5UEFGSET5U2UaVNV%1^=]MNO[M\=]:]!^^%=<=EnEEMM!)[ W3:;ad1Z=); ; wUWl CI .E     AmnqrwnxzGoG  8ESEX7X"%m"4SP}o|P}M})~~.@!DQ_QaCKCWCWX[C\]CAaQ7]?F?R?RSV?WX!Q3XHMHYHYZ]H^$_1a!%<$_GJ3PZP^P^`DEFQGHHadehjqarJMNQS]SaSab}~S@JAPSTWYcYgYghBCDYEPFUXY\^aaN@MaN^NUOgjkmoyo}o}~PQRoSgTU!d cp|A C  Otzz#E!UYZ[__jqr  %B   +-**5\*]^3A^^.. QR x||HLL9BBCUV VFIJJORXZacx{BDMORSXOY[^_k[lnqrn@BTRUVNNGbefobpJq]^1V9UVW^U_`abm`nopq|o}~AJKLMNVLWKXJqrJJ/0"**=S2TVXV`V`a~VBDBLBLMlBmJJ/0NNsfkulabwxyGwHIOPQRXPYZ[\gZhijk{i|}~H}IJKLWJXY]YcYcdefndo}B YC D E F P D Q R S T ^ R _ ` a b r ` s t u v G t H I J K ^ I _ ` a b l ` m n o p G n H I J K \ I ] ^ _ ` k ^ l m n o  m @ A B C Q A R S T U a S b c d e s c t u v w F u G H I J [ H \ ] ` a b c x a y ] z { ~ @ABR@S@W@WXd@e@kik p pq@ A ZEFEJEJKSET ZXZ{ [v\ulmNNAT`DaOPQ|_U[\]^k\lmnpAnBCDEUCVW[WaWabcdpbqDWEFJFPFPQRShQiw|F}~@H~IEJDabNNsQQS TXxy|~LzMyNNPWQU~L~MMOSPTjknotkuvCkCjDDFRGK`aderas`ttvIwy{}MOQRRE S-@-F-F-H IcaA3eA3b! IIE ||E V_`cdv`wxyVz {sqQR1#R' |#4Qy\$35G4HJ {{E""499U#3D#87"K] ^Ac!fH ^^]C_wWxatPya~b V^<RvJt(^X$_$_ JqulDa?qIam I { ^s. n) n.$ n3? n8 n=2oo9oo o o<o o%o  o&1o,' o2!o7 'C o< Ap 0!p p 83p pp8p p#ctj}|jdd|jdd|jdd|j}t t j t|dd t y) Nz--jsonzA/opt/struktur/knowledge-curator/minimal-human-review-dry-run.json)rz--sqlitez7/opt/struktur/knowledge-curator/minimal-human-review.dbz --markdownz?/opt/struktur/knowledge-curator/minimal-human-review-dry-run.mdFrFr) argparseArgumentParser add_argument parse_argsprintr)rrr)aprs rmainrs  "BBOOHGJO%KMOM\M\]gqjM\Mkmom|m|}ISTm|mU ==?DE$**WT]WXbf"ghr__main__)returnr)rrrr)rzset[str])rzargparse.Namespacerr)rNone)__doc__ __future__rrr)r# collectionsrrrrpathlibrminimal_review_corer r r r r rr%rrr0rr__name__rrrr s`M" ,'ii MN2L o^di  zFr