This tutorial shows how to implement an intelligence pipeline for documents using deepDoctection 1.2.x It combines the layout recognition, OCR and reading-order reconstruction with annotation linking. The analyzer is configured with DocLayNet based layout detection and Table Transformer structure identification, followed by DocTR OCR. We then examine the Page objects that result to better understand the way deepDoctection presents text, figures and tables as well as relationships, provenance and reading order. In addition, we extend the framework to include custom object types as well as our PipelineComponent that extracts monetary and dates entities and classifies documents based on their tabular properties. Finaly, we manually build a pipeline with ServiceFactory. We explore filtering, service rollback and serializing processed pages. Finally, document annotations are transformed into JSONL chunks for use in downstream RAG systems and retrieval.
Install!pip -q "deepdoctection" "transformers>=5.2.0" "timm" "python-doctr" "pdfplumber" "networkx" "lxml"
Import os
os.environ["DD_USE_TORCH"] = "True"
os.environ["DPI"] = "200"
os.environ["LOG_LEVEL"] = "INFO"
os.environ["ENABLE_DYNAMIC_OBJECT_TYPES"] = "False"
Textwrap, import json and re.
Import Path from pathlib
From Collections Import Counter
Numpy can be imported as np
Import matplotlib.pyplot into plt
Import HTML from IPython.display
import deepdoctection as dd
print("deepdoctection:", dd.__version__)
import transformers.integrations.peft as _hf_peft
if _hf_peft.is_peft_available():
_hf_peft.is_peft_available = lambda: False
print("patched: PEFT adapter lookup disabled for from_pretrained")
!mkdir -p /content/docs /content/imgs
!wget -q -O /content/docs/paper.pdf
!wget -q -O /content/imgs/finance.png
https://raw.githubusercontent.com/deepdoctection/notebooks/main/sample/finance/1bcac3899c9cb1c0b0f650b1431d3d52_7.png
The path of PDF is:"/content/docs/paper.pdf")
PNG = Path"/content/imgs/finance.png")
The path is OUT."/content/out"); OUT.mkdir(exist_ok=True)
Def show(img w=16),
If image is None: Return
plt.figure(figsize=(w, w * 1.3)); plt.axis("off"); plt.imshow(img); plt.show()
def analyze_any(pipe, path, **kw):
"""
Display correctly a directory or PDF file, as well as a single picture.
DoctectionPipe can stream a directory or a PDF from disk, but a *single*
image has no reader — path= only supplies the file name / provenance, and
Pixels must be sent in bytes=. You get the following:
When passing the path of a single picture, the bytes in that image are passed.
It must be done
"""
Path = path
If path.is_dir():
kw.setdefault("file_type", [".jpg", ".png", ".jpeg", ".tif"])
return pipe.analyze(path=path, **kw)
if path.suffix.lower() == ".pdf":
return pipe.analyze(path=path, **kw)
if path.suffix.lower() (".png", ".jpg", ".jpeg", ".tif"):
return pipe.analyze(path=path, bytes=path.read_bytes(), **kw)
raise ValueError(f"unsupported input: {path}")
Install the deepDoctection dependency, set up its runtime, and then apply a compatibility fix for Transformers or PEFT. The sample image and PDF files are downloaded and we prepare the output directory. Also, we define functions that help visualize images, analyze PDFs and image files, and perform a consistent analysis of directories and PDFs.
dd.print_model_infos(add_description=False, add_config=False, add_categories=False)
profile = dd.ModelCatalog.get_profile("Aryn/deformable-detr-DocLayNet/model.safetensors")
print("nlayout model categories:", profile.categories)
print("is registered:", dd.ModelCatalog.is_registered("Aryn/deformable-detr-DocLayNet/model.safetensors"))
config_overwrite = [
"USE_ROTATOR=False",
"USE_LAYOUT=True",
"USE_LAYOUT_NMS=True",
"USE_TABLE_SEGMENTATION=True",
"USE_TABLE_REFINEMENT=False",
"USE_PDF_MINER=False",
"USE_OCR=True",
"USE_LAYOUT_LINK=True",
"LAYOUT.WEIGHTS=Aryn/deformable-detr-DocLayNet/model.safetensors",
"ITEM.WEIGHTS=deepdoctection/tatr_tab_struct_v2/model.safetensors",
"ITEM.FILTER=['table']",
"OCR.USE_DOCTR=True",
"OCR.USE_TESSERACT=False",
"OCR.USE_TEXTRACT=False",
"OCR.WEIGHTS.DOCTR_WORD=doctr/db_resnet50/db_resnet50-ac60cadc.pt",
"OCR.WEIGHTS.DOCTR_RECOGNITION=doctr/crnn_vgg16_bn/crnn_vgg16_bn-0417f351.pt",
"SEGMENTATION.THRESHOLD_ROWS=0.4",
"SEGMENTATION.THRESHOLD_COLS=0.4",
"SEGMENTATION.FULL_TABLE_TILING=True",
"WORD_MATCHING.RULE=ioa",
"WORD_MATCHING.THRESHOLD=0.3",
"WORD_MATCHING.MAX_PARENT_ONLY=True",
"TEXT_ORDERING.INCLUDE_RESIDUAL_TEXT_CONTAINER=True",
"TEXT_ORDERING.PARAGRAPH_BREAK=0.035",
"TEXT_ORDERING.BROKEN_LINE_TOLERANCE=0.003",
"LAYOUT_LINK.PARENTAL_CATEGORIES=['figure','table']",
"LAYOUT_LINK.CHILD_CATEGORIES=['caption']",
]
analyzer = dd.get_dd_analyzer(config_overwrite=config_overwrite)
print("n--- pipeline ---")
for sid, name in analyzer.get_pipeline_info().items():
print(f"{sid} {name}")
print("n--- what this pipeline produces ---")
print(analyzer.get_meta_annotation())
The model registry of deepDoctection is checked to confirm the supported categories and layout models. Configuring the analyzer explicitly to include layout detection, DocTR OCR (word matching), reading-order reconstruction and layout linking. Then we initialize the analysis and examine its pipeline components as well as the annotation types it produces.
df = analyze_any(analyzer, PDF, session_id="tutorial01", max_datapoints=3)
df.reset_state()
Pages = list (df)
print(f"nparsed {len(pages)} pages")
The page is equal to pages[0]
show(page.viz(show_figures=True, show_residual_layouts=True, show_table_structure=True))
print("== narrative text ==")
print(textwrap.fill(page.text[:900], 110))
print("n== layout blocks in reading order ==")
For doc_id img_id pno ann_id order cat txt txt in the page.chunks[:12]:
print(f"[{order:>3}] {str(cat): caption ids:", [i[:8] for i in linked])
If page.words
Page.words = w[0]
print("nword:", w.characters, "| service:", w.service_id,
"| model:", w.model_id, "| bbox:", [round(x) for x in w.bbox])
tbl_pages = [p for p in pages if p.tables]
If tbl_pages
t = Tbl_pages[0].tables[0]
print(f"table {t.number_of_rows}x{t.number_of_columns}, "
The f"max_row_span={t.max_row_span}, max_col_span={t.max_col_span}")
display(HTML(t.html))
Row in the t.csv[:5]:
print([c[:22] "c" in row"
For c, use t.cells[:5]:
print(f" r{c.row_number} c{c.column_number} "
F"(span {c.row_span}x{c.column_span}) {c.text[:40]!r}")
else:
print("no table on these pages — the finance.png sample below has one")
The configured analyzer is run on the PDF sample and the pages are generated by the lazy data flow. Inspecting narrative text, chunks of reading, categories for annotations, relationships between figure captions and words, bounding boxes, and word provenance. We can also examine tables detected through HTML, CSV or individual cell representations.
@dd.object_types_registry.register("CustomKey")
class CustomKey(dd.ObjectTypes):
"""Custom summary keys — must be registered to be serialisable."""
MONEY_MENTIONS = "money_mentions"
DATE_MENTIONS = "date_mentions"
DOC_FLAVOUR = "doc_flavour"
@dd.object_types_registry.register("FlavourLabel")
class FlavourLabel(dd.ObjectTypes):
TABULAR = "tabular"
Narrative = "narrative"
MIXED = "mixed"
MONEY = (r.compile)"(?:[$€£]s?d[d,.]*|d[d,.]*s?(?:USD|EUR|GBP|million|bn))")
DATE = Recompile (r"b(?:d{1,2}[/-]d{1,2}[/-]d{2,4}|d{4}-d{2}-d{2}|"
The r"(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)w*s+d{1,2},?s+d{4})b")
class EntityAndFlavourService(dd.PipelineComponent):
def __init__(self, name="entity_flavour", tabular_ratio=0.25):
self.tabular_ratio = tabular_ratio
You can also check out our supersized().__init__(name)
def serve(self, dp: dd.Image) -> None:
page = dd.Page.from_image(dp, text_container=dd.LayoutLabel.WORD)
text = page.text_no_line_break
money = sorted(set(MONEY.findall(text)))
dates = sorted(set(DATE.findall(text)))
tables = page.tables
table_area = sum((b[2] The - b[0]) * (b[3] The - b[1](t.bbox in the tables for t)
ratio = table_area / float(page.width * page.height or 1)
flavor = (FlavourLabel.TABULAR if ratio > self.tabular_ratio
else FlavourLabel.NARRATIVE if not tables
Mixes (FlavourLabel)
self.dp_manager.set_summary_annotation(
summary_key=CustomKey.MONEY_MENTIONS, summary_name=CustomKey.MONEY_MENTIONS,
summary_value=money)
self.dp_manager.set_summary_annotation(
summary_key=CustomKey.DATE_MENTIONS, summary_name=CustomKey.DATE_MENTIONS,
summary_value=dates)
self.dp_manager.set_summary_annotation(
summary_key=CustomKey.DOC_FLAVOUR, summary_name=flavour,
summary_score=round(ratio, 4))
def clone(self):
return self.__class__(self.name, self.tabular_ratio)
def get_meta_annotation(self) -> dd.MetaAnnotation:
return dd.MetaAnnotation(
image_annotations=(),
sub_categories={},
relationships={},
summaries=(CustomKey.MONEY_MENTIONS, CustomKey.DATE_MENTIONS, CustomKey.DOC_FLAVOUR),
)
for k in (CustomKey.MONEY_MENTIONS, CustomKey.DATE_MENTIONS, CustomKey.DOC_FLAVOUR):
dd.Page.add_attribute_name(k)
Registering custom object types is used to classify document flavors, date and monetary references. Implementing a deepDoctection component, we analyze page text and coverage of tables to produce these page-level summary. We expose these custom summary field as Page attributes, so we can use them directly in processed documents.
from deepdoctection.analyzer import cfg, ServiceFactory
cfg.freeze(False)
cfg.USE_TABLE_SEGMENTATION = True
cfg.freeze(True)
Components = []
layout_detector = ServiceFactory.build_layout_detector(cfg, mode="LAYOUT")
components.append(ServiceFactory.build_layout_service(cfg, detector=layout_detector, mode="LAYOUT"))
components.append(ServiceFactory.build_layout_nms_service(cfg))
item_detector = ServiceFactory.build_layout_detector(cfg, mode="ITEM")
components.append(ServiceFactory.build_sub_image_service(cfg, detector=item_detector, mode="ITEM"))
components.append(ServiceFactory.build_table_segmentation_service(cfg, detector=item_detector))
word_detector = ServiceFactory.build_doctr_word_detector(cfg)
components.append(ServiceFactory.build_doctr_word_detector_service(word_detector))
components.append(ServiceFactory.build_text_extraction_service(cfg, ServiceFactory.build_ocr_detector(cfg)))
components.append(ServiceFactory.build_word_matching_service(cfg))
components.append(ServiceFactory.build_text_order_service(cfg))
components.append(EntityAndFlavourService())
custom_pipe = dd.DoctectionPipe(pipeline_component_list=components)
print("ncustom pipeline:", list(custom_pipe.get_pipeline_info().values()))
df2 = analyze_any(custom_pipe, PNG)
df2.reset_state()
fin_page = next(iter(df2))
print("flavour :", fin_page.doc_flavour)
print("money :", fin_page.money_mentions[:10])
print("dates :", fin_page.date_mentions[:10])
show(fin_page.viz(show_table_structure=True), w=13)
def skip_if_no_table(dp: dd.Image) -> bool:
You can return to your original language by clicking here. "table" not in {dp.get_annotation: a is not listed in the a.category_name.()}
components[-1].set_inbound_filter(skip_if_no_table)
det_sid = next(sid for sid, n in analyzer.get_pipeline_info().items()
If (n.startswith)"image_doctr"))
det_comp = analyzer.get_pipeline_component(service_id=det_sid)
df_undo = det_comp.undo(dd.DataFromList([p.base_image for p in pages]))
df_undo.reset_state()
undone = list(df_undo)
print("annotations before/after undo:",
len(pages[0].get_annotation()),
len(dd.Page.from_image(undone[0]).get_annotation()))
Manually, we assemble the deepDoctection Pipeline with ServiceFactory. We combine layout analysis and table processing along with OCR, ordering of text, and our own custom component. This custom pipeline is executed on the image of a financial document and we inspect detected flavors, monetary value, dates and table structures. Also, we apply an inbound filters and show how to undo annotations created by selected DocTR services.
For i and p, in the enumerate (pages):
p.save(image_to_json=False, path=OUT / f"page_{i}.json")
restored = dd.Page.from_file(str(OUT / "page_0.json"))
print("round-trip:", len(restored.get_annotation()), "of",
len(pages[0].get_annotation()), "annotations restored")
Record = []
For p, see pages
If you want to put doc_id and img_id in chunks, please use the following:
If txt.strip and txt.strip():
records.append({"document_id": doc_id, "page": pno, "order": order,
"category": str(cat), "annotation_id": ann_id, "text": txt})
For p.tables, t.
records.append({"document_id": p.document_id, "page": p.page_number,
"order": -1, "category": "table_html",
"annotation_id": t.annotation_id, "text": t.html})
(OUT / "chunks.jsonl").write_text("n"Joining records using.join (json.dumps.r for the rth record)
print(f"n{len(records)} chunks -> {OUT/'chunks.jsonl'}")
print(json.dumps(records[0], indent=2)[:400])
Serializing each page processed to JSON preserves its annotations and structural information without embedding original image data. Reloading a page saved in the database and comparing annotation counts allows us to confirm that structural data survived serialization. Finally, we convert narrative chunks, table HTML, and other HTML records into JSONL files that can be used directly by RAG and retrieval pipelines, as well as downstream document processing.
As a conclusion, we gained a better understanding of how deepDoctection orchestrates a variety of document-analysis services. In addition to running an analyzer, we also examined model registrations and controlled individual services. We also extracted tables, created custom metadata summary, and composed our own pipeline stages. We examined the effect of service filters and undos on annotations. This gave us a finer level of control over document processing workflows. Finally, serializing the structure of the processed documents was done. RAG chunks were generated, providing a base for AI document applications such as document search, knowledge extract, retrieval and augmented generation.
Click here to find out more FULL CODES here. Also, feel free to follow us on Twitter Join our Facebook group! 150k+ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.
Want to promote your GitHub repo, Hugging Face page, Product release or Webinar?? Connect with us
Sana Hassan is a dual-degree IIT Madras student and consulting intern with Marktechpost. She loves to apply technology and AI in order to solve real-world problems. He has a passion for solving real-world problems and brings an innovative perspective at the intersection between AI and practical solutions.

