ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
Kyungmin
Min
1
,
Minbeom
Kim
1
,
Kang-il
Lee
2
,
Seunghyun
Yoon
3
,
Kyomin
Jung
1
,
2
⋆
1
IPAI,
Seoul
National
University
2
Dept.
of
ECE,
Seoul
National
University
3
Adobe
Research
{kyungmin97,
kjung}@snu.ac.kr
Abstract.
Detailed
image
captioning
demands
both
factual
ground-
ing
and
fine-grained
coverage,
yet
existing
methods
have
struggled
to
achieve
them
simultaneously.
We
address
this
tension
with
Reflective
Note-Guided Captioning (ReflectCAP), where a multi-agent pipeline an-
alyzes
what
the
target
large
vision-language
model
(LVLM)
consistently
hallucinates
and
what
it
systematically
overlooks,
distilling
these
pat-
terns
into
reusable
guidelines
called
Structured
Reflection
Notes.
At
in-
ference time, these notes steer the captioning model along both axes—what
to
avoid
and
what
to
attend
to—yielding
detailed
captions
that
jointly
improve factuality and coverage. Applying this method to 8 LVLMs span-
ning
the
GPT-4.1
family,
Qwen
series,
and
InternVL
variants,
Reflect-
CAP
reaches
the
Pareto
frontier
of
the
trade-off
between
factuality
and
coverage,
and
delivers
substantial
gains
on
CapArena-Auto,
where
gen-
erated captions are judged head-to-head against strong reference models.
Moreover,
ReflectCAP
offers
a
more
favorable
trade-off
between
caption
quality
and
compute
cost
than
model
scaling
or
existing
multi-agent
pipelines, which incur 21–36% greater overhead. This makes high-quality
detailed
captioning
viable
under
real-world
cost
and
latency
constraints.
Keywords:
Detailed
Image
Captioning
·
Large
Vision-Language
Mod-
els
·
Multi-Agent
Systems
1
Introduction
Hyper-detailed captions
capture not only salient objects but also their attributes,
orientations,
spatial
relations,
background
context,
and
subtle
visual
states,
forming a comprehensive textual representation of an image. Such captions have
become a key ingredient in downstream multimodal systems—improving prompt
fidelity for text-to-image and text-to-video generation, and serving as a reasoning
aid for compositional and grounded decision-making [1,2,8,9,12,24]. Large vision-
language
models
(LVLMs)
can
produce
these
descriptions
fluently
[5,20,21,41],
yet
they
frequently
hallucinate—generating
text
that
is
not
grounded
in
the
⋆
Corresponding
authors
arXiv:2604.12357v1 [cs.AI] 14 Apr 2026
2
Min
et
al.
Fig. 1:
Overview
of
ReflectCAP.
In
the
offline
phase,
a
multi-agent
reflective
learning
pipeline
distills
a
target
LVLM’s
recurring
captioning
errors
and
omissions
into
Struc-
tured
Reflection
Notes.
In
the
online
phase,
these
notes
guide
caption
generation
for
new
images,
producing
captions
that
better
balance
factuality
and
coverage.
image.
This
limitation
is
widely
attributed
to
the
tendency
of
language
priors
to
progressively
dominate
over
visual
evidence
as
generation
length
increases,
leading
the
model
to
describe
what
is
statistically
probable
rather
than
what
is
actually
depicted
[15,16,19,25].
Hyper-detailed
captioning,
which
inherently
requires such extended generation, thus remains a critical bottleneck for reliable
deployment.
The straightforward remedy is supervised fine-tuning on human-authored de-
tailed
captions
[8,26]
to
improve
LVLMs’
intrinsic
performance;
however,
as
we
demonstrate
in
Section
5.1,
such
captions
often
exceed
the
model’s
perceptual
capacity,
even
increasing
hallucinations
well
beyond
the
base
model.
Moreover,
this approach requires not only expensive human annotation but also additional
training,
further
limiting
its
practicality.
An
alternative
is
inference-time
cor-
rection,
where
the
model
iteratively
revises
its
own
output
without
additional
training—an
approach
that
has
proven
effective
for
LLMs
[13, 22].
However,
recent
studies
show
that
LVLMs
struggle
to
self-correct
during
inference
with-
out
external
feedback,
as
they
tend
to
confirm
rather
than
rectify
their
own
errors
[10, 38].
Moreover,
iterative
revision
lengthens
the
text
context,
further
amplifying
language-prior
reliance
over
visual
evidence
[16,25].
Taken
together,
these
limitations
suggest
that
a
single
LVLM
alone
is
unlikely
to
fully
address
the
detail–faithfulness
tension,
and
that
external
guidance
from
a
multi-agent
system
is
needed.
To
this
end,
we
propose
ReflectCAP
(Reflective
Note-Guided
Captioning),
a
gradient-free
framework
that
distills
a
target
LVLM’s
recurring
errors
into
an
agentic
memory
called
Structured
Reflection
Notes
and
leverages
them
for
inference-time
steering
(Figure
1).
ReflectCAP
operates
in
two
distinct
phases.
In
the
offline
phase,
a
multi-agent
pipeline
critiques
the
target
model’s
captions
against
a
small
set
of
human-annotated
references
to
diagnose
its
systematic
error
patterns,
separately
encoding
1)
hallucination
patterns
and
2)
omis-
sion
patterns
into
generalized
guideline
notes.
In
the
online
phase,
each
set
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
3
of
notes
serves
a
distinct
role
in
steering
generation:
one
produces
a
grounded
base
caption
that
suppresses
recurring
hallucinations,
and
the
other
produces
a
detail-focused caption covering typically neglected visual elements. A final merge
step
combines
both
with
the
image
as
a
reference,
producing
a
comprehensive
caption
that
improves
factuality
and
coverage
simultaneously.
Empirically, ReflectCAP sets new pareto frontier under the factuality–coverage
evaluation
proposed
by
Lee
et
al.
[16],
substantially
expanding
coverage
while
preserving
precision
and
achieving
the
highest
F1
across
model
families.
This
improvement
also
holds
on
CapArena-Auto
[3],
a
pairwise
benchmark
that
eval-
uates
captions
holistically
and
is
closely
aligned
with
human
preference.
On
600
images
evaluated
with
CapArena-Auto,
ReflectCAP
yields
substantial
win-rate
improvements
of
+32.2
and
+21.9
points
over
the
corresponding
baselines
for
the
GPT
and
open-source
model
families,
respectively.
Furthermore,
Reflect-
CAP
can
elevate
smaller
models
beyond
flagship
baselines;
for
instance,
GPT-
4.1-mini
with
our
method
surpasses
GPT-5.2
in
CapArena-Auto
win
rate.
Be-
yond performance comparisons, ReflectCAP consistently offers a more compute-
efficient path to improving caption quality than either scaling model size or scal-
ing
inference-time
computation;
for
example,
InternVL3.5-4B
with
ReflectCAP
achieves
a
comparable
factuality–coverage
F1
to
InternVL3.5-38B
while
requir-
ing
approximately
8
times
lower
inference
TFLOPs,
and
outperforms
existing
multi-agent
pipelines
while
incurring
21–36%
lower
compute
overhead,
suggest-
ing
that
ReflectCAP
can
serve
as
a
practical,
cost-effective
alternative
to
both
model
scaling
and
inference-time
computation
scaling
for
detailed
captioning.
2
Related
Work
2.1
Detailed
Image
Captioning
Obtaining
large-scale
detailed
image
captions
is
increasingly
important,
as
they
serve
as
training
signals
for
text-to-image
and
text-to-video
generation
and
as
reasoning
aids
for
compositional
vision–language
tasks
[1,2,8,9].
Approaches
to
scaling detailed captions broadly follow two directions. The first relies on human-
authored
dense
captions
(e.g.,
DCI
[33],
DOCCI
[26],
IIW
[8]),
which
offer
high
fidelity
and
coverage
but
are
prohibitively
expensive
to
scale
beyond
limited
datasets.
The
second
approach
employs
LVLMs
as
automated
captioners.
How-
ever,
in
long-form
generation,
these
models
often
over-rely
on
language
priors,
which results in hallucinated but unsupported details [15,25] and the omission of
subtle
visual
attributes
[7, 23, 28].
Many
existing
mitigation
methods
primarily
improve
factuality
(precision)
without
comparably
improving
descriptive
cov-
erage
(recall),
leaving
the
two
principal
quality
axes
of
detailed
captioning
in
tension
[6,11,17,35,40,42].
Our work addresses this trade-off by distilling reflec-
tion
notes
that
explicitly
capture
recurring
hallucination
patterns
and
missing-
detail patterns, enabling separate control of hallucination suppression and detail
recovery
at
inference
time.
4
Min
et
al.
2.2
Reflective
Memory
in
Agentic
Frameworks
Recent advancements in LLM-based agents have successfully leveraged reflective
memory
to
refine
behavior
in
long-horizon
tasks.
By
analyzing
historical
trajec-
tories
and
inference-time
feedback,
these
text-based
models
synthesize
reusable
reasoning
strategies
and
deploy
them
at
appropriate
moments
to
guide
subse-
quent
multi-step
decision-making
[27,30,32,34,43].
However, extending this long-horizon reflection paradigm to LVLMs is funda-
mentally
problematic.
Unlike
pure
text
generation,
LVLMs
struggle
to
reliably
extract
error
patterns
over
extended
reasoning
chains
[10, 38].
As
the
number
of
inference
steps
increases,
the
visual
evidence
becomes
increasingly
diluted,
and
the
models
fall
prey
to
severe
language
prior
phenomena—relying
more
on
text-induced
hallucinations
than
on
the
grounded
visual
input
[4,18,25,31]
To
address
these
intrinsic
limitations,
we
shift
from
online,
long-horizon
tra-
jectory
tracking
to
an
offline,
bottom-up
distillation
process.
By
systematically
aggregating image-specific feedback across diverse samples, we identify and distill
the
target
LVLM’s
recurring
hallucination
and
omission
patterns
into
a
gener-
alized
reflection
memory.
This
distilled
memory
is
then
utilized
to
proactively
steer
the
model
during
inference,
bypassing
the
need
for
costly
step-by-step
re-
finement.
3
Reflective
Note-Guided
Captioning
Framework
We
introduce
Reflective
Note-Guided
Captioning
(ReflectCAP),
a
framework
that 1) distills systematic error patterns of a target LVLM into compact, reusable
directives, and 2) leverages them as guidance to improve both factuality and de-
tailedness in image captioning. The key insight is that large vision-language mod-
els exhibit predictable failure modes, including recurring hallucinations and con-
sistent
blind
spots.
Once
surfaced,
these
patterns
can
be
counteracted
through
lightweight prompt-level intervention rather than costly retraining or multi-agent
inference
at
test
time.
ReflectCAP
operationalizes
this
insight
in
two
stages:
an
offline
phase
that analyzes a small exemplar set through a multi-agent pipeline
to
construct
structured
reflection
notes
encoding
the
model’s
characteristic
er-
rors, and an
online phase
that injects these notes into the generation context for
new
images,
achieving
improved
factuality
and
coverage
at
negligible
additional
cost.
Figure
2
illustrates
the
overall
pipeline.
3.1
Offline
Phase:
Constructing
Structured
Reflection
Notes
The
goal
of
the
offline
phase
is
to
surface
the
target
LVLM’s
systematic
error
patterns
and
encode
them
as
reusable
guidance.
Given
a
small
exemplar
set
D
train
=
{
(
x
i
, y
∗
i
)
}
M
i
=1
of
images
paired
with
human-written
reference
captions,
where
x
i
denotes
the
i
-th
input
image,
y
∗
i
its
corresponding
reference
caption,
and
M
the
total
number
of
exemplars,
we
run
a
three-agent
pipeline
that
pro-
gressively
moves
from
raw
errors
to
generalizable
directives.
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
5
Fig. 2:
ReflectCAP
framework.
In
the
offline
phase
,
a
multi-agent
pipeline
analyzes
a
small
exemplar
set
to
distill
recurring
errors
and
omissions
of
the
target
LVLM
into
Structured Reflection Notes
. In the
online phase
, these notes guide caption generation:
Avoid
Notes
suppress
hallucinations,
Include
Notes
encourage
missing
details,
and
a
final
merge
integrates
grounded
and
detail-focused
captions
into
the
final
output.
Captioning
Agent.
The
pipeline
begins
by
letting
the
target
LVLM
caption
each
image
x
i
in
a
zero-shot
manner,
producing
a
candidate
caption
ˆ
y
i
with
no
additional
guidance.
This
is
intentional:
the
resulting
captions
faithfully
re-
flect the model’s default behavior—including its characteristic hallucinations and
omissions—providing
an
unbiased
basis
for
the
diagnosis
that
follows.
Feedback
Agent.
Each candidate caption is then critiqued against two sources
of evidence: the image itself and the human reference. The Feedback Agent cross-
references
ˆ
y
i
against
both
x
i
and
y
∗
i
,
producing
a
structured
issue
report
I
i
for
each
example.
Reports
are
organized
into
two
categories:
–
Hallucinations
:
details
in
ˆ
y
i
that
are
factually
incorrect
or
not
visible
in
x
i
.
–
Missing
Details
:
important
details
present
in
y
∗
i
that
are
absent
from
ˆ
y
i
.
For
example,
a
hallucination
might
be
“the
caption
states
two
people
are
sitting,
but
the
image
shows
three,”
while
a
missing
detail
might
be
“the
caption
does
not
mention
the
wooden
railing
visible
in
the
foreground.”
The
Feedback
Agent
has
access
to
the
image
during
critique,
ensuring
that
its
judgments
are
visually
grounded.
However,
at
this
stage,
every
diagnosis
is
tied
to
a
particular
image
and
caption,
making
it
difficult
to
apply
directly
at
inference
time.
Note
Organizer.
Instance-specific
diagnoses
help
explain
individual
failures,
but
reusable
guidance
requires
identifying
patterns
—errors
that
recur
across
images.
The
Note
Organizer
performs
this
consolidation.
Because
diagnoses
col-
lected
across
images
can
easily
exceed
the
LVLM
context
window,
the
organizer
processes
them
incrementally,
consuming
batches
and
updating
a
running
set
of
6
Min
et
al.
Algorithm
1
Offline:
Constructing
Structured
Reflection
Notes
Require:
Training
set
D
train
=
{
(
x
i
, y
∗
i
)
}
M
i
=1
,
max
items
K
,
batch
size
B
1:
N
←∅
2:
for
i
= 1
to
M
do
3:
ˆ
y
i
←
CaptioningAgent
(
x
i
)
▷
Zero-shot
captioning
4:
I
i
←
FeedbackAgent
(
x
i
,
ˆ
y
i
, y
∗
i
)
▷
Instance-specific
critique
5:
end
for
6:
for
each
batch
B
⊂{I
1
, . . . ,
I
M
}
of
size
B
do
7:
N
←
NoteOrganizer
(
B
,
N
, K
)
▷
Cross-instance
generalization
8:
end
for
9:
return
N
= (
N
avoid
,
N
include
)
Algorithm
2
Online:
Note-Steered
Caption
Generation
Require:
Test
image
x
,
Structured
Reflection
Notes
N
= (
N
avoid
,
N
include
)
1:
Step
1:
Grounded
Base
Caption
2:
c
base
←
VLM
(
x,
N
avoid
)
▷
Suppress
known
hallucination
patterns
3:
Step
2:
Detail-Focused
Caption
4:
c
detail
←
VLM
(
x,
N
include
)
▷
Attend
to
typically
neglected
details
5:
Step
3:
Merging
Distinct
Captions
6:
c
final
←
VLM
(
x,
c
base
,
c
detail
)
▷
Merge
with
image
as
reference
7:
return
c
final
notes
after
each
step.
During
each
update,
it
merges
semantically
similar
issues
and
abstracts
them
into
broadly
applicable
rules.
By
imposing
an
upper
bound
of
K
items,
the
note
set
retains
only
patterns
corresponding
to
frequently
re-
curring
mistakes
or
commonly
omitted
details,
thereby
pruning
redundant
or
overly
narrow
entries.
The
result
is
a
compact,
prioritized
set
of
notes
that
we
call
Structured
Reflection
Notes
.
It
consists
of
two
complementary
compo-
nents:
–
Avoid
Notes
N
avoid
:
directives
that
suppress
recurrent
hallucination
pat-
terns
(e.g.,
“Do
not
infer
object
colors
when
they
are
ambiguous”
).
–
Include
Notes
N
include
:
directives
that
enforce
frequently
omitted
details
(e.g.,
“Describe
visible
architectural
details
such
as
structural
supports
and
railings”
).
This
progression
from
instance-level
diagnosis
to
cross-instance
generalization
is
what
allows
the
notes
to
capture
the
model’s
systematic
tendencies
rather
than
one-off
mistakes.
In
practice,
M
=30
exemplar
images
and
K
=5
items
per
category
are
sufficient,
as
shown
in
our
ablation
study
(§5.2).
Algorithm
1
sum-
marizes
the
full
offline
procedure.
3.2
Online
Phase:
Note-Steered
Caption
Generation
Once constructed, the Structured Reflection Notes replace the multi-agent pipeline
entirely. Given a new image
x
and the pre-computed notes
N
= (
N
avoid
,
N
include
)
,
the
online
phase
generates
a
caption
through
at
most
three
LVLM
calls.
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
7
Step
1:
Grounded
Base
Caption.
N
avoid
is
injected
into
the
captioning
prompt, directing the LVLM to suppress its known hallucination patterns during
generation.
This
produces
a
grounded
base
caption
c
base
that
is
more
factually
reliable
than
a
zero-shot
caption
while
preserving
the
model’s
natural
descrip-
tive
ability.
Since
this
step
requires
exactly
one
forward
pass,
almost
identical
in
cost
to
zero-shot
inference,
it
can
serve
as
a
standalone
variant
for
captioning
pipelines
where
factuality
is
the
primary
concern.
Step
2:
Detail-Focused
Caption.
A
second
call
uses
N
include
to
direct
the
LVLM’s attention toward the specific types of details it typically misses, such as
material
textures,
background
elements,
and
spatial
arrangements,
producing
a
detail-focused
caption
c
detail
.
This
caption
captures
the
descriptive
details
that
c
base
trades
off
in
favor
of
factual
grounding.
Step
3:
Merging
Distinct
Captions.
The
final
stage
merges
c
base
and
c
detail
into a unified caption
c
final
, using the image as a grounding reference. To prevent
the
integration
of
spurious
details,
the
merge
strategy
is
conservative:
c
base
is
treated
as
the
primary
source
of
truth,
while
c
detail
serves
as
a
supplementary
source.
In
cases
of
conflict,
the
model
prioritizes
c
base
,
as
it
is
generated
under
hallucination-suppressing
guidance
designed
for
factual
grounding.
We
refer
to
this
full
three-step
pipeline
as
ReflectCAP
.
The
complete
procedure
is
sum-
marized
in
Algorithm
2.
4
Experiments
We
evaluate
ReflectCAP
along
three
dimensions.
(1)
Fine-grained
evalua-
tion
: fine-grained factuality and coverage, which examines the trade-off between
these
two
axes,
(2)
Holistic
evaluation
:
holistic
caption
quality,
which
tests
whether fine-grained gains translate into perceived overall quality, and (3)
Cost-
efficiency
:
computational
cost
analysis
which
examines
whether
the
method
is
practical
enough
for
downstream
deployment.
4.1
Experimental
Settings
Evaluation
Metrics.
We
adopt
two
complementary
evaluation
suites,
both
well-aligned
with
human
judgments.
For
fine-grained
evaluation,
we
evaluate
on
IIW-400
dataset
[8]
using
the
factuality
and
coverage
metrics
proposed
by
Lee
et
al.
[16].
Factuality
(Precision)
decomposes
each
caption
into
atomic
proposi-
tions and verifies each against the image and ground-truth;
Coverage
(Recall) is
measured via curated VQA items associated with each IIW-400 image, answered
using
only
the
generated
caption.
We
report
F1
score
as
the
harmonic
mean
of
factuality
and
coverage.
For
holistic
evaluation,
we
adopt
CapArena-Auto
[3],
a
pairwise
benchmark
scored
by
average
win-rate
margin
against
three
reference
models.
Baseline
LVLMs.
We evaluate across eight LVLMs spanning closed-source and
open-source families at a range of scales: two proprietary models (GPT-4.1-mini,
GPT-4.1-nano)
and
six
open-weight
instruction-tuned
models,
comprising
three
8
Min
et
al.
Table
1:
Factuality
and
Coverage
on
IIW-400.
Precision
measures
the
ratio
of
verified-true
propositions.
Recall
measures
the
ratio
of
correctly
answered
VQA
questions.
F1
is
the
harmonic
mean.
Best
per
model
in
bold
.
∆
denotes
F1
change
from
Zero-shot.
Model
Method
P
R
F1
∆
Model
Method
P
R
F1
∆
GPT-4.1-mini
Zero-shot
83.6
68.1
75.1
—
Qwen2.5-7B
Zero-shot
68.3
57.9
62.7
—
Few-shot
81.9
71.5
76.3
+1.2
Few-shot
49.9
56.6
53.1
−
9.6
Self-Corr.
82.6
69.3
75.4
+0.3
Self-Corr.
63.8
57.3
60.4
−
2.3
CapMAS
84.2
67.7
75.1
0.0
CapMAS
72.0
57.4
63.9
+1.2
ReflectCAP
83.8
72.0
77.5
+2.4
ReflectCAP
68.8
62.3
65.4
+2.7
GPT-4.1-nano
Zero-shot
78.9
62.6
69.8
—
Qwen2.5-32B
Zero-shot
71.9
64.0
67.7
—
Few-shot
77.1
67.1
71.8
+2.0
Few-shot
64.9
64.5
64.7
−
3.0
Self-Corr.
79.2
62.6
70.0
+0.2
Self-Corr.
70.2
66.2
68.1
+0.4
CapMAS
83.1
57.2
67.8
−
2.0
CapMAS
73.5
63.1
67.9
+0.2
ReflectCAP
78.0
67.2
72.2
+2.4
ReflectCAP
69.9
66.4
68.1
+0.4
InternVL-4B
Zero-shot
64.1
54.9
59.1
—
Qwen3-8B
Zero-shot
76.3
69.2
72.6
—
Few-shot
49.4
52.8
51.1
−
8.0
Few-shot
71.3
72.2
71.8
−
0.8
Self-Corr.
62.1
54.1
57.8
−
1.3
Self-Corr.
76.3
70.0
73.0
+0.4
CapMAS
72.2
53.3
61.3
+2.2
CapMAS
79.3
68.8
73.7
+1.1
ReflectCAP
64.8
61.3
63.0
+3.9
ReflectCAP
77.0
71.2
74.0
+1.4
InternVL-38B
Zero-shot
72.5
57.8
64.3
—
Qwen3-32B
Zero-shot
79.2
72.5
75.7
—
Few-shot
63.2
62.1
62.6
−
1.7
Few-shot
73.4
74.4
73.9
−
1.8
Self-Corr.
73.7
58.2
65.0
+0.7
Self-Corr.
78.6
73.6
76.0
+0.3
CapMAS
78.6
57.5
66.4
+2.1
CapMAS
81.0
72.0
76.2
+0.5
ReflectCAP
73.7
64.2
68.6
+4.3
ReflectCAP
79.2
73.8
76.4
+0.7
smaller
models
(InternVL3.5-4B-Instruct,
Qwen2.5-VL-7B-Instruct, Qwen3-VL-
8B-Instruct)
and
three
larger
models
(InternVL3.5-38B-Instruct,
Qwen2.5-VL-
32B-Instruct,
Qwen3-VL-32B-Instruct).
Baseline
Methods.
For each model, we compare ReflectCAP against four cap-
tioning
strategies.
Zero-shot
uses
a
minimal
prompt
(“Describe
this
image
in
detail”).
Few-shot
prepends
three
randomly
sampled
human-annotated
caption
exemplars.
Self-Correction
first
generates
a
zero-shot
caption,
then
revises
it
af-
ter
re-examining
the
image.
CapMAS
[16],
a
multi-agent
baseline,
decomposes
a
caption
into
atomic
propositions
via
specialized
agents,
verifies
each
against
the
image,
and
rewrites
the
caption
by
removing
unverified
content.
For
fair
comparison,
ReflectCAP
uses
the
target
model
itself
for
both
the
offline
phase
(note
construction)
and
the
online
phase
(caption
generation),
ensuring
that
no
external
model
contributes
to
the
final
output.
In
the
offline
phase,
we
use
im-
ages
and
reference
captions
from
IIW-Eval
that
are
not
included
in
IIW-400
to
construct
the
notes.
4.2
Fine-Grained
Evaluation
Table 1 presents our main results. Existing methods tend to improve one axis at
the cost of the other: few-shot prompting increases coverage by imitating human-
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
9
Table 2: CapArena-Auto scores.
Score denotes the average win-rate margin (higher
is
better;
range
[
−
100
,
100]
).
For
CapMAS
and
ReflectCAP,
we
report
score
with
the
change relative to zero-shot in parentheses. Rows marked with
†
use zero-shot reference
values
taken
from
the
provided
CapArena
caption.
Model
Zero-shot
CapMAS
ReflectCAP
(Ours)
Leaderboard
anchors
(zero-shot
only)
GPT-5.2
70.0
—
—
Gemini-1.5-Pro
†
62.3
—
—
GPT-4o-0806
†
44.3
—
—
Qwen2.5VL-72B
†
39.7
—
—
Claude-3.5-Sonnet
†
29.7
—
—
GPT-4.1-mini
57.7
53.7
(
−
4
.
0
)
90.0
(
+32
.
3
)
GPT-4.1-nano
21.2
−
14
.
7
(
−
35
.
9
)
51.3
(
+30
.
1
)
InternVL3.5-4B
−
54
.
3
−
46
.
7
(
+7
.
6
)
−
18
.
7
(
+35
.
6
)
InternVL3.5-38B
−
24
.
0
−
15
.
0
(
+9
.
0
)
12.0
(
+36
.
0
)
Qwen2.5-VL-7B
−
33
.
7
−
28
.
0
(
+5
.
7
)
−
7
.
3
(
+26
.
4
)
Qwen2.5-VL-32B
5.7
8.0
(
+2
.
3
)
25.3
(
+19
.
6
)
Qwen3-VL-8B
76.0
74.0
(
−
2
.
0
)
85.3
(
+9
.
3
)
Qwen3-VL-32B
87.3
87.0
(
−
0
.
3
)
91.7
(
+4
.
4
)
authored demonstrations but pushes the model beyond its perceptual boundary,
causing
factuality
to
drop.
Self-correction
yields
only
marginal
gains
regardless
of
model
scale,
indicating
that
models
struggle
to
identify
and
fix
errors
in
their
own
generated
captions
through
revision
alone.
CapMAS
that
improves
factu-
ality
by
removing
unreliable
content
from
existing
captions
inevitably
sacrifices
coverage
in
the
process.
In
contrast,
ReflectCAP
substantially
boosts
coverage
while
incurring
minimal
loss
in
factuality,
achieving
the
highest
F
1
across
all
eight models. This demonstrates that the Structured Reflection Notes effectively
balance
the
inherent
trade-off:
coverage
guidance
encourages
the
model
to
de-
scribe more, which inevitably risks lowering factuality, while factuality guidance
separately constrains this degradation. By controlling each objective through its
own
dedicated
guidance,
ReflectCAP
achieves
the
highest
F
1
across
all
models,
advancing
the
Pareto
frontier
between
the
two
objectives.
4.3
Holistic
Evaluation
Fine-grained
metrics
measure
factuality
and
coverage
in
isolation,
but
it
is
also
necessary
to
evaluate
overall
caption
quality.
CapArena-Auto
(Table
2)
tests
this
by
pitting
each
method’s
captions
against
fixed
three
reference
models
4
in
head-to-head
comparisons
judged
by
GPT-4.1-mini.
ReflectCAP
improves
the
average
win-rate
margin
by
+33.2
points
for
the
GPT
family
and
+21.9
points
for
open-source
models
over
their
zero-shot
base-
lines, confirming that the fine-grained gains in §4.2 translate directly into holistic
4
GPT-4o-0806,
CogVLM2-llama3-chat-19B,
and
MiniCPM-V2.6-8B.
10
Min
et
al.
10
2
TFLOPs / Final Caption
57.5
60.0
62.5
65.0
67.5
70.0
72.5
75.0
77.5
F1 Score
InternVL3.5-4B
InternVL3.5-38B
Qwen2.5VL-7B
Qwen2.5VL-32B
Qwen3VL-8B
Qwen3VL-32B
InternVL3.5
Qwen2.5VL
Qwen3VL
Zero-shot
ReflectCAP
CapMAS
Zero-shot
→
ReflectCAP
Zero-shot
→
CapMAS
Scaling (Small
→
Large)
Fig. 3:
Solid
and
dash-dotted
lines
denote improvements from zero-shot
to Reflect-
CAP and CapMAS, respectively. ReflectCAP achieves higher F1 scores while requiring
21–36%
less
compute
than
CapMAS.
Light
dashed
lines
denote
performance
gains
from
model
parameter
scaling.
Compared
to
simply
increasing
model
size,
ReflectCAP
achieves
comparable
quality
at
up
to
8
×
lower
compute
cost,
enabling
high-quality,
detailed
captioning
more
practical
under
real-world
cost
and
latency
constraints.
caption
quality.
Notably,
GPT-4.1-mini
with
ReflectCAP
(90.0)
surpasses
even
GPT-5.2
(70.0),
suggesting
that
structured
reflection
notes
can
compensate
for
inherent
model
capacity
differences
in
overall
caption
quality.
Additionally,
Re-
flectCAP
yields
consistent
positive
gains
over
the
zero-shot
baseline
across
all
models, whereas CapMAS tends to degrade performance when applied to models
that
already
exhibit
strong
zero-shot
capabilities.
4.4
Cost-Efficiency
Beyond
caption
quality,
practical
deployment
requires
efficient
inference.
To
ex-
amine
this,
we
measure
cost-efficiency
across
methods
and
open-source
models
in
terms
of
the
total
TFLOPs
required
to
produce
a
final
caption
at
inference
time.
5
Figure
3
plots
F
1
against
inference
TFLOPs
per
final
caption
for
all
open-
source
models.
Two
trends
emerge.
First,
ReflectCAP
improves
caption
quality
more compute-efficiently than simply scaling model size. For example, InternVL3.5-
4B
with
ReflectCAP
achieves
an
F
1
of
63.0,
approaching
InternVL3.5-38B
zero-
shot
(64.3)
while
requiring
roughly
7.8×
fewer
TFLOPs
(27.5
vs.
213.7).
Simi-
larly,
Qwen3-VL-8B
with
ReflectCAP
maintains
a
comparable
performance
gap
relative to Qwen3-VL-32B zero-shot, while generating captions at approximately
5
We
approximate
inference
cost
as
C
≈
2
NT
,
where
N
is
the
number
of
non-
embedding
parameters
and
T
is
the
total
token
count.
For
methods
with
multiple
calls
per
image,
image
tokens
are
counted
only
once
via
KV
caching.
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
11
Table
3:
Impact
of
SFT
on
detailed
captioning
factuality.
Model
Method
Fact.
Cov.
F1
InternVL3.5-4B
Zero-shot
64.1
54.9
59.1
SFT
w/
Human-authored
58.2
56.5
57.4
SFT
w/
ReflectCAP-generated
66.5
61.6
63.9
Qwen2.5-VL-7B
Zero-shot
68.3
57.9
62.7
SFT
w/
Human-authored
57.1
58.0
57.6
SFT
w/
ReflectCAP-generated
69.9
63.5
66.5
3.3× lower computational cost. Second, ReflectCAP is also more compute-efficient
than existing inference-time baselines within the same model. Across three mod-
els— InternVL3.5-4B, Qwen2.5-VL-7B, and Qwen3-VL-8B—ReflectCAP reduces
inference
cost
by
21%–36%
TFLOPs
compared
to
CapMAS,
while
consistently
achieving
higher
F
1
scores.
This
inefficiency
stems
from
CapMAS
applying
its
multi-agent
pipeline
directly
at
inference
time,
and
its
focus
on
factuality
alone
limits
overall
caption
quality.
These
results
highlight
ReflectCAP
as
a
practical
alternative
to
both
model
scaling
and
compute-heavy
inference-time
pipelines.
5
Analysis
5.1
Supervised
Fine-Tuning
on
Detailed
Image
Captioning
The most intuitive approach to improving detailed captioning is supervised fine-
tuning
(SFT)
on
human-authored
detailed
captions.
To
examine
this,
we
fine-
tune
InternVL3.5-4B
and
Qwen2.5-VL-7B
on
9,647
DOCCI
human-annotated
captions using LoRA. As shown in Table 3, SFT substantially degrades factuality
compared to the zero-shot baseline for both models (evaluation follows the same
protocol as Section 4), lending further support to recent findings that training on
annotations exceeding the model’s visual capabilities amplifies hallucinations [36,
37].
An
interesting
finding
is
that
replacing
human
captions
with
ReflectCAP-
generated captions on the same DOCCI images for training maintains factuality
while improving coverage. This suggests that ReflectCAP can serve as a scalable
pipeline
for
constructing
training
data
that
improves
recall
while
staying
within
the
model’s
visual
boundary,
without
manual
annotation
effort.
5.2
Ablation
Study
Effect
of
Grounded
Base
Caption.
As
shown
in
Figure
4,
the
Grounded
Base
Caption,
which
applies
only
the
hallucination-suppression
notes,
improves
factuality
over
the
zero-shot
baseline
in
nearly
all
models.
However,
the
magni-
tude of this improvement varies with the model’s instruction-following capability.
Models
with
stronger
instruction-following
abilities,
such
as
GPT-4.1-mini
and
GPT-4.1-nano,
generate
well-grounded
base
captions
where
hallucination
pat-
terns are effectively suppressed, whereas the Qwen2.5-VL family shows marginal
12
Min
et
al.
GPT-4.1
mini
GPT-4.1
nano
Qwen3-VL
32B
Qwen3-VL
8B
Qwen2.5-VL
32B
Qwen2.5-VL
7B
InternVL3.5
38B
InternVL3.5
4B
55
60
65
70
75
80
85
90
Factuality (%)
83.6
78.9
79.2
76.3
71.9
68.3
72.5
64.1
85.9
80.2
79.7
77.3
69.3
68.5
74.1
64.2
Zero-shot
Grounded base caption
Fig. 4:
Factuality
comparison
between
Zero-shot
and
Grounded
Base
Caption
across
all
models.
Models
with
stronger
instruction-following
capabilities
show
larger
gains.
Table
4:
Separate
vs.
Combined
note
injection.
Separate-Merge
applies
Avoid
and
Include
notes
in
separate
passes
with
merging;
Combined
injects
both
into
a
single
prompt.
Model
Method
Fact.
Cov.
F1
GPT-4.1-mini
Separate-Merge
84.8
72.8
78.3
Combined
84.8
70.7
77.1
GPT-4.1-nano
Separate-Merge
78.1
68.6
73.1
Combined
78.4
67.0
72.2
Qwen3-VL-8B
Separate-Merge
77.6
72.4
74.9
Combined
74.4
71.2
72.7
InternVL3.5-4B
Separate-Merge
66.6
62.4
64.4
Combined
55.8
58.8
57.3
improvement
or
even
degradation
compared
to
the
zero-shot
baseline.
This
sug-
gests
that
as
instruction-following
capabilities
of
LVLMs
continue
to
advance,
ReflectCAP can achieve even greater improvements without any modification to
the
framework.
Separate
vs.
Combined
Injection.
Table
4
compares
two
strategies
on
100
images
sampled
from
IIW-400:
applying
Avoid
and
Include
notes
in
separate
generation
passes
versus
injecting
both
into
a
single
prompt.
Across
all
four
models,
the
separated
approach
consistently
outperforms
the
combined
variant.
This
effect
is
particularly
pronounced
in
InternVL3.5-4B,
where
the
combined
injection
causes
a
severe
F1
drop
(64.4
→
57.3),
indicating
that
overloading
the
prompt
with
too
many
directives
can
be
detrimental,
especially
for
models
with
limited instruction-following capability. These results confirm that hallucination
suppression
and
detail
recovery
are
better
handled
as
separate
objectives.
Number
of
Exemplars
and
Note
Items.
Figure
5
analyzes
two
key
param-
eters
of
the
offline
phase
using
100
images
sampled
from
IIW-400
for
efficient
evaluation.
For
the
number
of
exemplar
images
N
(Figure
5(a)),
all
models
show
a
substantial
improvement
from
zero-shot
to
N
=
10
,
with
performance
largely
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
13
0
(zero-shot)
10
30
100
Number of Exemplar Images (
N
)
0.625
0.650
0.675
0.700
0.725
0.750
0.775
0.800
F1 Score
(a)
GPT-4.1-mini
InternVL3.5-38B
Qwen3-VL-8B
0
(zero-shot)
1
5
10
Number of Note Items per Category (
K
)
(b)
GPT-4.1-mini
InternVL3.5-38B
Qwen3-VL-8B
Fig. 5:
Ablation
on
note
construction
parameters.
(a)
F1
vs.
the
number
of
exemplar
images
N
.
Performance
saturates
at
N
≈
30
,
indicating
that
systematic
error
patterns
can
be
surfaced
from
a
modest
exemplar
set.
(b)
F1
vs.
the
maximum
number
of
note
items
per
category
K
.
Even
K
=1
already
yields
strong
gains,
with
performance
improving
slightly
further
at
K
=5
.
plateauing
around
N
=
30
and
slightly
declining
at
N
=
100
.
Qualitative
anal-
ysis
of
GPT-4.1-mini
suggests
that
larger
exemplar
pools
shift
reflection
notes
from
model-specific
guidance
toward
generic
instructions.
For
example,
notes
generated
with
N
= 30
contain
targeted
rules
such
as
“Do
not
add
unsupported
details
to
signs,
logos,
or
symbols,”
whereas those from
N
= 100
become broader
directives like
“Avoid
subjective
or
interpretive
descriptions
not
clearly
supported
by
the
image
or
reference.”
This
indicates
that
systematic
error
patterns
can
be
reliably
surfaced
from
a
modest
exemplar
set,
while
larger
pools
introduce
vari-
ation
that
dilutes
the
corrective
signal.
For
the
maximum
items
per
category
K
(Figure
5(b)),
even
K
=1
provides
a
substantial performance gain. Under this setting, models tend to produce a single
reflection note that aggregates multiple corrective signals rather than a narrowly
scoped
rule.
For
example,
a
reflection
note
generated
at
K
=1
states:
“Include
precise
visible
details
of
object
features,
spatial
distributions,
lighting
and
shadow
effects,
background
elements,
and
compositional
angles
to
ensure
completeness
and
accuracy.”
As
K
increases
(e.g.,
K
=5
),
these
aggregated
instructions
are
decomposed
into
multiple
more
specialized
reflection
notes
that
introduce
more
specific
guidelines,
which
further
improves
performance
over
the
K
=1
setting.
While
some
models
continue
improving
up
to
K
=10
,
others
peak
around
K
=5
and slightly decline thereafter, suggesting that the optimal number of guidelines
varies
across
models
and
may
depend
on
their
ability
to
incorporate
multiple
instructions.
5.3
Case
Study
Figure
6
illustrates
a
case
study
examining
each
pipeline
stage
of
ReflectCAP
using
GPT-4.1
mini.
In
the
zero-shot
caption,
hallucinations
are
observed
in
fine-grained
details
such
as
roof
shape,
window
count
and
arrangement,
and
14
Min
et
al.
Fig. 6:
Case
study
of
our
pipeline.
Top
:
Zero-shot
Caption
Middle
:
ReflectCAP-
Base
suppresses
hallucinations
via
Avoid
notes.
Bottom
:
ReflectCAP-Full
recovers
embossed
text
details
guided
by
Include
notes.
Red
denotes
hallucinated
expressions,
blue
denotes
hallucination-corrected
descriptions,
and
green
denotes
recovered
fine-
grained
details.
sign appearance. By applying our hallucination avoidance patterns, ReflectCAP
corrects
the
window
count
from
six
to
five
and
omits
unverifiable
arrangements,
yielding
a
more
factually
grounded
description.
Furthermore,
the
missing
de-
tail
recovery
patterns
enable
the
model
to
capture
previously
overlooked
visual
elements,
including
the
lion’s
head
fountain
spout
and
cast
shadows.
These
re-
sults
demonstrate
that
ReflectCAP
effectively
encodes
common
error
patterns
into
structured
reflection
notes
and
leverages
them
to
steer
caption
generation
toward
greater
accuracy
and
visual
fidelity.
6
Conclusion
We presented ReflectCAP, a tuning-free framework that distills a target LVLM’s
recurring
hallucination
and
omission
patterns
into
Structured
Reflection
Notes.
By
leveraging
these
notes
at
caption
generation
time,
ReflectCAP
steers
the
model separately for each pattern type—suppressing hallucinations and recover-
ing missing details—then merges the resulting captions into a single, comprehen-
sive
description.
Across
8
LVLMs,
ReflectCAP
consistently
achieves
the
highest
F1
score
on
factuality–coverage
evaluation,
and
substantially
outperforms
all
baselines
on
CapArena-Auto.
Furthermore,
from
a
compute-efficiency
perspec-
tive,
ReflectCAP
is
more
effective
than
both
scaling
up
model
size
and
scaling
inference-time
computation
for
improving
caption
quality.
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
15
References
1.
Betker,
J.,
Goh,
G.,
Jing,
L.,
Brooks,
T.,
Wang,
J.,
Li,
L.,
Ouyang,
L.,
Zhuang,
J.,
Lee, J., Guo, Y., et al.: Improving image generation with better captions. Computer
Science.
https://cdn.
openai.
com/papers/dall-e-3.
pdf
2
(3),
8
(2023)
2.
Brooks,
T.,
Peebles,
B.,
Holmes,
C.,
DePue,
W.,
Guo,
Y.,
Jing,
L.,
Schnurr,
D.,
Taylor,
J.,
Luhman,
T.,
Luhman,
E.,
et
al.:
Video
generation
models
as
world
simulators.
OpenAI
Blog
1
(8),
1
(2024)
3.
Cheng, K., Song, W., Fan, J., Ma, Z., Sun, Q., Xu, F., Yan, C., Chen, N., Zhang, J.,
Chen, J.: CapArena: Benchmarking and analyzing detailed image captioning in the
LLM
era.
In:
Che,
W.,
Nabende,
J.,
Shutova,
E.,
Pilehvar,
M.T.
(eds.)
Findings
of
the
Association
for
Computational
Linguistics:
ACL
2025.
pp.
14077–14094.
Association
for
Computational
Linguistics,
Vienna,
Austria
(Jul
2025).
https:
//doi.org/10.18653/v1/2025.findings-acl.724
,
https://aclanthology.org/
2025.findings-acl.724/
4.
Chung, J., Kim, J., Kim, S., Lee, J., Kim, M.S., Yu, Y.: v1: Learning to point visual
tokens for multimodal grounded reasoning (2026),
https://arxiv.org/abs/2505.
18842
5.
Dai,
W.,
Li,
J.,
Li,
D.,
Tiong,
A.,
Zhao,
J.,
Wang,
W.,
Li,
B.,
Fung,
P.N.,
Hoi,
S.:
Instructblip:
Towards
general-purpose
vision-language
models
with
instruction
tuning. Advances in neural information processing systems
36
, 49250–49267 (2023)
6.
Favero,
A.,
Zancato,
L.,
Trager, M.,
Choudhary,
S., Perera, P., Achille, A.,
Swami-
nathan,
A.,
Soatto,
S.:
Multi-modal
hallucination
control
by
visual
information
grounding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern
Recognition.
pp.
14303–14312
(2024)
7.
Fu,
X.,
Hu,
Y.,
Li,
B.,
Feng,
Y.,
Wang,
H.,
Lin,
X.,
Roth,
D.,
Smith,
N.A.,
Ma,
W.C.,
Krishna,
R.:
Blink:
Multimodal
large
language
models
can
see
but
not
per-
ceive.
In:
European
Conference
on
Computer
Vision.
pp.
148–166.
Springer
(2024)
8.
Garg,
R.,
Burns,
A.,
Karagol
Ayan,
B.,
Bitton,
Y.,
Montgomery,
C.,
Onoe,
Y.,
Bunner,
A.,
Krishna,
R.,
Baldridge,
J.M.,
Soricut,
R.:
ImageInWords:
Unlocking
hyper-detailed
image
descriptions.
In:
Al-Onaizan,
Y.,
Bansal,
M.,
Chen,
Y.N.
(eds.)
Proceedings
of
the
2024
Conference
on
Empirical
Methods
in
Natural
Lan-
guage
Processing.
pp.
93–127.
Association
for
Computational
Linguistics,
Miami,
Florida,
USA
(Nov
2024).
https://doi.org/10.18653/v1/2024.emnlp-main.6
,
https://aclanthology.org/2024.emnlp-main.6/
9.
Gutflaish,
E.,
Kachlon,
E.,
Zisman,
H.,
Hacham,
T.,
Sarid,
N.,
Visheratin,
A.,
Huberman,
S.,
Davidi,
G.,
Bukchin,
G.,
Goldberg,
K.,
et
al.:
Generating
an
im-
age
from
1,000
words:
Enhancing
text-to-image
with
structured
captions.
arXiv
preprint
arXiv:2511.06876
(2025)
10.
He,
J.,
Lin,
H.,
Wang,
Q.,
Fung,
Y.R.,
Ji,
H.:
Self-correction
is
more
than
refine-
ment:
A
learning
framework
for
visual
and
language
reasoning
tasks.
In:
Findings
of the Association for Computational Linguistics: ACL 2025. pp. 6405–6421 (2025)
11.
Huang,
Q.,
Dong,
X.,
Zhang,
P.,
Wang,
B.,
He,
C.,
Wang,
J.,
Lin,
D.,
Zhang,
W.,
Yu,
N.:
Opera:
Alleviating
hallucination
in
multi-modal
large
language
models
via
over-trust
penalty
and
retrospection-allocation.
In:
Proceedings
of
the
IEEE/CVF
Conference
on
Computer
Vision
and
Pattern
Recognition.
pp.
13418–13427
(2024)
12.
Ju,
X.,
Gao,
Y.,
Zhang,
Z.,
Yuan,
Z.,
Wang,
X.,
Zeng,
A.,
Xiong,
Y.,
Xu,
Q.,
Shan, Y.: Miradata: A large-scale video dataset with long durations and structured
captions.
Advances
in
Neural
Information
Processing
Systems
37
,
48955–48970
(2024)
16
Min
et
al.
13.
Kamoi, R., Zhang, Y., Zhang, N., Han, J., Zhang, R.: When can LLMs actually cor-
rect
their
own
mistakes?
a
critical
survey
of
self-correction
of
LLMs.
Transactions
of
the
Association
for
Computational
Linguistics
12
,
1417–1440
(2024).
https://
doi.org/10.1162/tacl_a_00713
,
https://aclanthology.org/2024.tacl-1.78/
14.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray,
S., Radford, A., Wu, J., Amodei, D.: Scaling laws for neural language models. arXiv
preprint
arXiv:2001.08361
(2020)
15.
Lee,
K.i.,
Kim,
M.,
Yoon,
S.,
Kim,
M.,
Lee,
D.,
Koh,
H.,
Jung,
K.:
VLind-bench:
Measuring
language
priors
in
large
vision-language
models.
In:
Chiruzzo,
L.,
Rit-
ter,
A.,
Wang,
L.
(eds.)
Findings
of
the
Association
for
Computational
Linguis-
tics:
NAACL
2025.
pp.
4129–4144.
Association
for
Computational
Linguistics,
Albuquerque,
New
Mexico
(Apr
2025).
https://doi.org/10.18653/v1/2025.
findings-naacl.231
,
https://aclanthology.org/2025.findings-naacl.231/
16.
Lee,
S.,
Yoon,
S.,
Bui,
T.,
Shi,
J.,
Yoon,
S.:
Toward
robust
hyper-detailed
image
captioning:
A
multiagent
approach
and
dual
evaluation
metrics
for
factuality
and
coverage.
In:
Forty-second
International
Conference
on
Machine
Learning
(2025),
https://openreview.net/forum?id=REnIf3dCsI
17.
Leng,
S.,
Zhang,
H.,
Chen,
G.,
Li,
X.,
Lu,
S.,
Miao,
C.,
Bing,
L.:
Mitigating
object
hallucinations in large vision-language models through visual contrastive decoding.
In:
Proceedings
of
the
IEEE/CVF
Conference
on
Computer
Vision
and
Pattern
Recognition.
pp.
13872–13882
(2024)
18.
Li,
Z.,
Shi,
H.,
Gao,
Y.,
Liu,
D.,
Wang,
Z.,
Chen,
Y.,
Liu,
T.,
Zhao,
L.,
Wang,
H.,
Metaxas,
D.N.:
The
hidden
life
of
tokens:
Reducing
hallucination
of
large
vision-
language
models
via
visual
information
steering.
In:
Forty-second
International
Conference
on
Machine
Learning
(2025),
https://openreview.net/forum?id=
7BKcLeHQsm
19.
Liu,
F.,
Lin,
K.,
Li,
L.,
Wang,
J.,
Yacoob,
Y.,
Wang,
L.:
Mitigating
hallucina-
tion
in
large
multi-modal
models
via
robust
instruction
tuning.
arXiv
preprint
arXiv:2306.14565
(2023)
20.
Liu,
H.,
Li,
C.,
Li,
Y.,
Lee,
Y.J.:
Improved
baselines
with
visual
instruction
tun-
ing.
In:
Proceedings
of
the
IEEE/CVF
conference
on
computer
vision
and
pattern
recognition.
pp.
26296–26306
(2024)
21.
Liu,
H.,
Li,
C.,
Wu,
Q.,
Lee,
Y.J.:
Visual
instruction
tuning.
Advances
in
neural
information
processing
systems
36
,
34892–34916
(2023)
22.
Madaan,
A.,
Tandon,
N.,
Gupta,
P.,
Hallinan,
S.,
Gao,
L.,
Wiegreffe,
S.,
Alon,
U.,
Dziri,
N.,
Prabhumoye,
S.,
Yang,
Y.,
Gupta,
S.,
Majumder,
B.P.,
Hermann,
K.,
Welleck,
S.,
Yazdanbakhsh,
A.,
Clark,
P.:
Self-refine:
Iterative
refinement
with
self-feedback
(2023),
https://arxiv.org/abs/2303.17651
23.
Marsili,
D.,
Mehta,
A.,
Lin,
R.Y.,
Gkioxari,
G.:
Same
or
not?
enhancing
visual
perception
in
vision-language
models.
arXiv
preprint
arXiv:2512.23592
(2025)
24.
Merchant,
N.,
de
Ocáriz
Borde,
H.S.,
Popescu,
A.C.,
Suarez,
C.G.J.:
Structured
captions improve prompt adherence in text-to-image models (re-laion-caption 19m)
(2025),
https://arxiv.org/abs/2507.05300
25.
Min,
K.,
Kim,
M.,
Lee,
K.i.,
Lee,
D.,
Jung,
K.:
Mitigating
hallucinations
in
large
vision-language models via summary-guided decoding. In: Chiruzzo, L., Ritter, A.,
Wang, L. (eds.) Findings of the Association for Computational Linguistics: NAACL
2025.
pp.
4183–4198.
Association
for
Computational
Linguistics,
Albuquerque,
New
Mexico
(Apr
2025).
https://doi.org/10.18653/v1/2025.findings-naacl.
235
,
https://aclanthology.org/2025.findings-naacl.235/
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
17
26.
Onoe,
Y.,
Rane,
S.,
Berger,
Z.,
Bitton,
Y.,
Cho,
J.,
Garg,
R.,
Ku,
A.,
Parekh,
Z.,
Pont-Tuset,
J.,
Tanzer,
G.,
Wang,
S.,
Baldridge,
J.:
Docci:
Descriptions
of
connected
and
contrasting
images
(2024),
https://arxiv.org/abs/2404.19753
27.
Ouyang,
S.,
Yan,
J.,
Hsu,
I.H.,
Chen,
Y.,
Jiang,
K.,
Wang,
Z.,
Han,
R.,
Le,
L.T.,
Daruki,
S.,
Tang,
X.,
Tirumalashetty,
V.,
Lee,
G.,
Rofouei,
M.,
Lin,
H.,
Han,
J.,
Lee,
C.Y.,
Pfister,
T.:
Reasoningbank:
Scaling
agent
self-evolving
with
reasoning
memory
(2025),
https://arxiv.org/abs/2509.25140
28.
Rahmanzadehgervi,
P.,
Bolton,
L.,
Taesiri,
M.R.,
Nguyen,
A.T.:
Vision
language
models
are
blind:
Failing
to
translate
detailed
visual
features
into
words.
arXiv
preprint
arXiv:2407.06581
(2024)
29.
Rostamzadeh,
N.,
Hosseini,
S.,
Boquet,
T.,
Stokowiec,
W.,
Zhang,
Y.,
Jauvin,
C.,
Pal,
C.:
Fashion-gen:
The
generative
fashion
dataset
and
challenge
(2018),
https://arxiv.org/abs/1806.08317
30.
Shinn,
N.,
Cassano,
F.,
Gopinath,
A.,
Narasimhan,
K.,
Yao,
S.:
Reflexion:
Lan-
guage
agents
with
verbal
reinforcement
learning.
Advances
in
neural
information
processing
systems
36
,
8634–8652
(2023)
31.
Sun,
H.L.,
Sun,
Z.,
Peng,
H.,
Ye,
H.J.:
Mitigating
visual
forgetting
via
take-along
visual
conditioning
for
multi-modal
long
CoT
reasoning.
In:
Che,
W.,
Nabende,
J.,
Shutova,
E.,
Pilehvar,
M.T.
(eds.)
Proceedings
of
the
63rd
Annual
Meeting
of
the
Association
for
Computational
Linguistics
(Volume
1:
Long
Papers).
pp.
5158–5171. Association for Computational Linguistics, Vienna, Austria (Jul 2025).
https://doi.org/10.18653/v1/2025.acl-long.257
,
https://aclanthology.
org/2025.acl-long.257/
32.
Tan, Z., Yan, J., Hsu, I.H., Han, R., Wang, Z., Le, L., Song, Y., Chen, Y., Palangi,
H.,
Lee,
G.,
et
al.:
In
prospect
and
retrospect:
Reflective
memory
management
for
long-term personalized dialogue agents. In: Proceedings of the 63rd Annual Meeting
of
the
Association
for
Computational
Linguistics
(Volume
1:
Long
Papers).
pp.
8416–8439
(2025)
33.
Urbanek,
J.,
Bordes,
F.,
Astolfi,
P.,
Williamson,
M.,
Sharma,
V.,
Romero-Soriano,
A.:
A
picture
is
worth
more
than
77
text
tokens:
Evaluating
clip-style
models
on
dense
captions
(2024),
https://arxiv.org/abs/2312.08578
34.
Wan, G., Ling, M., Ren, X., Han, R., Li, S., Zhang, Z.: Compass: Enhancing agent
long-horizon
reasoning
with
evolving
context
(2025),
https://arxiv.org/abs/
2510.08790
35.
Wang, X., Pan, J., Ding, L., Biemann, C.: Mitigating hallucinations in large vision-
language
models
with
instruction
contrastive
decoding.
In:
Findings
of
the
Asso-
ciation
for
Computational
Linguistics:
ACL
2024.
pp.
15840–15853
(2024)
36.
Yanuka,
M.,
Ben-Kish,
A.,
Bitton,
Y.,
Szpektor,
I.,
Giryes,
R.:
Bridging
the
visual
gap:
Fine-tuning
multimodal
models
with
knowledge-adapted
captions.
In:
Chiruzzo,
L.,
Ritter,
A.,
Wang,
L.
(eds.)
Proceedings
of
the
2025
Confer-
ence
of
the
Nations
of
the
Americas
Chapter
of
the
Association
for
Compu-
tational
Linguistics:
Human
Language
Technologies
(Volume
1:
Long
Papers).
pp.
10497–10518.
Association
for
Computational
Linguistics,
Albuquerque,
New
Mexico
(Apr
2025).
https://doi.org/10.18653/v1/2025.naacl-long.527
,
https://aclanthology.org/2025.naacl-long.527/
37.
Yue,
Z.,
Zhang,
L.,
Jin,
Q.:
Less
is
more:
Mitigating
multimodal
hallucination
from
an
EOS
decision
perspective.
In:
Ku,
L.W.,
Martins,
A.,
Srikumar,
V.
(eds.)
Proceedings
of
the
62nd
Annual
Meeting
of
the
Association
for
Computational
Linguistics
(Volume
1:
Long
Papers).
pp.
11766–11781.
Association
for
Computa-
tional
Linguistics,
Bangkok,
Thailand
(Aug
2024).
https://doi.org/10.18653/
v1/2024.acl-long.633
,
https://aclanthology.org/2024.acl-long.633/
18
Min
et
al.
38.
Zhang,
L.,
Zeng,
X.,
Li,
K.,
Yu,
G.,
Chen,
T.:
Sc-captioner:
Improving
image
captioning
with
self-correction
by
reinforcement
learning.
In:
Proceedings
of
the
IEEE/CVF International Conference on Computer Vision. pp. 23145–23155 (2025)
39.
Zheng,
Y.,
Zhang,
R.,
Zhang,
J.,
Ye,
Y.,
Luo,
Z.:
Llamafactory:
Unified
efficient
fine-tuning
of
100+
language
models.
In:
Proceedings
of
the
62nd
annual
meeting
of the association for computational linguistics (volume 3: system demonstrations).
pp.
400–410
(2024)
40.
Zhou,
Y.,
Cui,
C.,
Yoon,
J.,
Zhang,
L.,
Deng,
Z.,
Finn,
C.,
Bansal,
M.,
Yao,
H.:
Analyzing
and
mitigating
object
hallucination
in
large
vision-language
models.
arXiv
preprint
arXiv:2310.00754
(2023)
41.
Zhu,
D.,
Chen,
J.,
Shen,
X.,
Li,
X.,
Elhoseiny,
M.:
Minigpt-4:
Enhancing
vision-
language
understanding
with
advanced
large
language
models.
arXiv
preprint
arXiv:2304.10592
(2023)
42.
Zhu,
L.,
Ji,
D.,
Chen,
T.,
Xu,
P.,
Ye,
J.,
Liu,
J.:
Ibd:
Alleviating
hallucinations
in
large
vision-language
models
via
image-biased
decoding.
In:
Proceedings
of
the
Computer
Vision
and
Pattern
Recognition
Conference.
pp.
1624–1633
(2025)
43.
Zhu,
X.,
Cai,
Y.,
Liu,
Z.,
Zheng,
B.,
Wang,
C.,
Ye,
R.,
Chen,
J.,
Wang,
H.,
Wang,
W.C., Zhang, Y., et al.: Toward ultra-long-horizon agentic science: Cognitive accu-
mulation for machine learning engineering. arXiv preprint arXiv:2601.10402 (2026)
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
19
Appendix
A
Supervised
Fine-tuning
Details
We
fine-tune
InternVL3.5-4B
and
Qwen2.5-VL-7B
on
captions
corresponding
to
the
DOCCI
images.
The
captions
come
from
two
sources:
the
original
human-
authored
captions
provided
in
DOCCI
and
captions
generated
by
our
Reflect-
CAP
pipeline.
We
apply
LoRA
(rank
=
64
,
α
=
128
,
dropout
=
0
.
05
)
to
all
linear
layers
of
the
language
model.
Training
runs
for
3
epochs
with
a
batch
size
of
2
per
device
and
4
gradient
accumulation
steps,
using
a
learning
rate
of
1
×
10
−
4
with cosine scheduling and a 3% warmup ratio. All models are trained in
BF16
precision
on
two
NVIDIA
A6000
GPUs
using
the
LLaMA-Factory
frame-
work
[39].
For
consistency,
we
report
results
from
the
final
checkpoint
after
3
training
epochs
for
all
supervised
fine-tuning
experiments.
Per-epoch
evaluation
results
are
reported
in
Table
5;
evaluation
follows
the
same
protocol
as
in
Sec-
tion
4.
Table
5:
Per-epoch
evaluation
results
for
fine-tuned
models
on
ReflectCAP-generated
and
human-authored
captions.
Model
Train
Data
Epoch
Precision
Recall
F1
Qwen2.5-VL-7B
ReflectCAP
1
0.7112
0.6228
0.6674
2
0.7065
0.6284
0.6651
3
0.6993
0.6346
0.6654
Human-authored
1
0.5702
0.5567
0.5634
2
0.5689
0.5754
0.5721
3
0.5715
0.5798
0.5756
InternVL3.5-4B
ReflectCAP
1
0.6885
0.6097
0.6467
2
0.6943
0.6085
0.6486
3
0.6649
0.6157
0.6393
Human-authored
1
0.5940
0.5602
0.5766
2
0.5763
0.5669
0.5716
3
0.5821
0.5653
0.5736
B
Prompt
Templates
We
report
here
the
full
prompt
templates
used
in
the
offline
and
online
stages
of
our
framework.
The
offline
templates
define
the
architectural
roles
of
the
three
agents—the
Captioning
Agent,
Feedback
Agent,
and
Note
Organizer—
which
collaborate
to
construct
the
Error
Notes
from
training
data.
The
online
templates are used at inference time to generate grounded base captions, extract
commonly
missed
details,
and
merge
the
two
into
a
final
refined
caption.
20
Min
et
al.
B.1
Offline
Stage
Prompts
Captioning
Agent.
System:
You
are
an
expert
image
captioner.
Describe
images
accurately
and
in
detail.
User:
Describe
this
image
in
detail.
Feedback
Agent.
System:
You
are
a
caption
quality
monitor.
Compare
the
generated
caption
with
the
reference
caption.
Your
task:
1.
Identify
HALLUCINATIONS
:
details
in
the
generated
caption
that
are
WRONG
or
NOT
visible
in
the
image.
2.
Identify
MISSING
DETAILS
:
important
details
in
the
reference
caption
that
are
MISSING
from
the
generated
caption.
For
each
issue,
provide:
(1)
what
the
issue
is,
(2)
why
it’s
problematic,
(3)
a
simple
rule
to
avoid/fix
it.
Output
format:
Hallucinations:
-
issue
1,
-
issue
2,
...
Missing
Details:
-
issue
1,
-
issue
2,
...
If
no
issues
are
found
in
a
category,
write
“None”.
User:
Generated
Caption:
{generated_caption}
Reference
Caption:
{reference_caption}
Analyze
the
generated
caption
against
the
reference
and
the
image.
Note
Organizer.
System:
You
manage
“Error
Notes”
for
an
image
captioning
model.
Your
task:
1.
Review
new
issues
from
this
batch.
2.
Update
the
error
notes
by
adding
new
issues,
merging
similar
ones,
summa-
rizing
into
general
rules,
and
keeping
maximum
k
items
per
category.
3.
Each
item
should
be
simple
and
compact
(one
line).
Output
format:
[Hallucination
-
Avoid
These]:
-
item
1,
-
item
2,
...
(max
k
)
[Missing
Detail
-
Include
These]:
-
item
1,
-
item
2,
...
(max
k
)
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
21
User:
Current
Error
Notes:
{current_notes}
New
Issues
from
Batch:
{batch_issues}
Update
the
Error
Notes.
Keep
it
compact
(max
k
items
per
category).
B.2
Online
Stage
Prompts
Stage
1:
Grounded
Base
Caption.
System:
You
are
an
expert
image
captioner.
When
describing
the
image,
avoid
these
common
errors:
{hallucination_notes}
.
Output
only
the
caption.
User:
Describe
this
image
in
detail.
Stage
2:
Detail-Focused
Caption.
System:
You
are
an
expert
image
captioner.
Describe
the
image
focusing
on
the
aspects
listed
below,
which
are
commonly
overlooked.
Only
describe
what
is
CLEARLY
VISIBLE
—
do
not
guess
or
infer.
Output
only
the
caption.
User:
Describe
this
image,
paying
special
attention
to
these
commonly
missed
aspects:
{missing_detail_notes}
Stage
3:
Merging
Distinct
Captions.
System:
You
supplement
a
base
caption
with
new
information
from
a
second
caption.
Rules:
–
The
base
caption
is
the
foundation
—
preserve
its
wording,
counts,
colors,
and
positions
as-is.
–
From
the
second
caption,
only
add
objects
or
elements
NOT
already
mentioned
in
the
base.
–
Do
NOT
change
any
existing
descriptions
(counts,
colors,
spatial
terms,
materials).
–
Verify
each
new
element
against
the
image
before
adding
it.
–
If the second caption has no genuinely new elements, return the base caption
unchanged.
User:
Base
caption:
{Grounded
Base
Caption}
Second
caption:
{Detail-Focused
Caption}
Add
only
new,
verified
elements
from
the
second
caption
into
the
base.
Do
not
modify
existing
details.
Output
only
the
final
caption:
22
Min
et
al.
C
Inference
Cost
Details
Beyond
caption
quality,
practical
deployment
also
requires
efficient
inference.
To
evaluate
this
aspect,
we
measure
cost-efficiency
across
different
methods
and
open-source models in terms of TFLOPs per image. Following Kaplan et al. [14],
we
approximate
the
inference
cost
of
a
single
forward
pass
as
C
≈
2
NT
,
where
N
denotes
the
number
of
non-embedding
parameters
and
T
is
the
total
number
of
processed
tokens.
For
methods
that
involve
multiple
calls
per
image,
image
tokens are counted only once through KV caching. As shown in Table 6, Reflect-
CAP
consistently
achieves
lower
inference
cost
than
CapMAS
across
all
evalu-
ated open-source models, while maintaining stronger captioning performance. In
particular,
ReflectCAP
requires
21–36%
less
compute
than
CapMAS.
Table
6:
Compute
cost
of
different
methods
on
open-source
models,
measured
in
TFLOPs
per
final
caption
with
KV
caching.
Model
Method
TFLOPs/final
caption
Qwen3-VL-32B
Zero-shot
219.97
CapMAS
490.83
ReflectCAP
356.48
Qwen3-VL-8B
Zero-shot
53.22
CapMAS
105.36
ReflectCAP
67.60
Qwen2.5-VL-32B
Zero-shot
280.00
CapMAS
533.68
ReflectCAP
385.41
Qwen2.5-VL-7B
Zero-shot
64.04
CapMAS
96.41
ReflectCAP
76.06
InternVL3.5-38B
Zero-shot
213.71
CapMAS
367.68
ReflectCAP
297.31
InternVL3.5-4B
Zero-shot
22.37
CapMAS
36.90
ReflectCAP
27.51
D
Effect
of
Note
Generator
We investigate whether the quality of
Structured Reflection Notes
improves when
a
more
capable
model
serves
as
the
note
author.
We
compare
two
conditions:
(1)
Self-generated
, where the same model serves as the captioning agent, feed-
back
agent,
and
note
organizer—i.e.,
the
target
model
analyzes
its
own
errors
and writes the error note itself; and (2)
GPT-4.1
mini
(proxy
writer)
, where
the
captioning
agent
remains
the
target
model
but
GPT-4.1
mini
replaces
both
the
feedback
agent
and
the
note
organizer,
analyzing
the
target
model’s
error
patterns
and
writing
the
note
on
its
behalf.
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
23
Table
7:
Self-generated
error
notes
vs.
notes
written
by
GPT-4.1
mini
as
a
proxy
writer.
Both note
types describe the target model’s error patterns; only the authorship
differs.
Bold
indicates
the
better
score
per
target
model.
Target
Model
Note
Source
Fact.
Cov.
F1
∆
F1
Qwen3-VL-8B
Self-generated
77.6
72.4
74.9
—
GPT-4.1
mini
76.7
73.3
74.9
±
0.0
Qwen3-VL-32B
Self-generated
77.9
75.7
76.8
—
GPT-4.1
mini
81.0
75.4
78.1
+1.3
Qwen2.5-VL-7B
Self-generated
68.1
63.5
65.7
—
GPT-4.1
mini
67.4
64.8
66.0
+0.3
Qwen2.5-VL-32B
Self-generated
73.9
68.6
71.2
—
GPT-4.1
mini
73.0
70.6
71.8
+0.6
InternVL3.5-4B
Self-generated
66.6
62.4
64.4
—
GPT-4.1
mini
65.2
62.2
63.7
−
0.7
InternVL3.5-38B
Self-generated
75.7
66.2
70.6
—
GPT-4.1
mini
72.7
66.8
69.6
−
1.0
Table
7
shows
that
replacing
the
note
author
with
a
more
capable
model
does
not
uniformly
improve
performance.
While
Qwen3-VL-32B
sees
a
notable
gain
(+1.3
F1),
the
remaining
configurations
show
only
marginal
improvements
or
even
slight
degradation,
particularly
in
the
InternVL3.5
family.
A qualitative comparison of the generated notes offers a possible explanation.
GPT-4.1
mini
tends
to
produce
general
principle-level
guidelines
such
as
“
Avoid
speculative
or
inferred
details
about
materials,
styles,
or
dates
,”
whereas
self-
generated
notes
are
more
case-specific,
e.g.,
“
Do
not
exaggerate
water
clarity
or
infer
bottom
composition
(e.g.,
‘sandy/silty’).
”
This
suggests
that
each
model
may
respond
better
to
a
particular
note
style,
and
that
a
universally
stronger
author
does
not
guarantee
a
better-fitting
note.
E
Qualitative
Analysis
Figures 7 and 8 present two contrasting examples on Qwen3-VL-8B that together
illustrate the strengths and limitations of ReflectCAP. In both cases, error notes
successfully
suppress
zero-shot
hallucinations:
the
grounded
base
caption
cor-
rects
a
fabricated
indentation
count
and
a
food
misidentification
in
Figure
7,
and
removes
an
invented
coat
pattern
and
a
reversed
body
orientation
in
Fig-
ure
8.
The
difference
emerges
when
merging
the
detail
caption
into
the
base
caption.
In
Figure
7,
the
note
prompts
the
model
to
describe
overhead
lighting
and
silverware
placement,
and
the
model
accurately
incorporates
these
details,
improving
coverage
without
sacrificing
factuality.
In
contrast,
Figure
8
shows
that
the
same
type
of
note
(e.g., “
Confirm
exact
placement
via
spatial
anchors
”)
instead
leads
to
a
fabricated
height
comparison
between
the
chicken
and
the
24
Min
et
al.
Zero-shot
caption
✗
“
each
dish
contains
nine
circular
inden-
tations
”
✗
“
a
soft,
custard-like
substance
topped
with
green
herb
garnish
”
Grounded
base
caption
✓
“
baked
escargot
”
(correct
food
identification)
✓
No
count
claim
—
avoids
fabrication
ReflectCAP
+
“
Shadows
from
overhead
lighting
fall
across
the
table
”
✓
+
“
A
fork
and
knife
rest
beside
the
lower
ramekin
”
✓
Error
notes
triggered:
Mention
visible
lighting
positions
;
Confirm
exact
placement
via
spatial
anchors
—
both
within
the
target
model’s
capa-
bility.
Fig. 7:
Success
case.
Error
notes
correct
zero-shot
hallucinations
(
red
→
green
),
and
the
extract-merge
step
successfully
adds
verifiable
details
(
blue
,
✓
).
goat.
This
illustrates
that
reflection
notes
can
guide
the
model
to
attend
to
pre-
viously
overlooked
details,
but
whether
this
results
in
faithful
descriptions
or
additional
hallucinations
depends
on
the
model’s
perceptual
ability.
Currently,
verifying
the
factuality
of
newly
added
details
is
left
to
the
model
itself
during
the
merging
step,
which
proves
insufficient
when
the
model
lacks
the
visual
understanding
ability
to
accurately
perceive
the
prompted
content.
A
more
explicit
verification
mechanism
at
this
stage
could
make
ReflectCAP
a
more
robust
framework
that
effectively
boosts
both
coverage
and
factuality.
F
Domain-Specific
Captioning
with
ReflectCAP
The
main
experiments
focus
on
detailed
captioning
of
everyday
images,
but
Re-
flectCAP’s Structured Reflection Notes are not tied to everyday image captioning—
they
adapt
automatically
to
the
exemplar
set
provided
in
the
offline
phase.
To
verify
this,
we
apply
ReflectCAP
to
fashion
product
captioning,
which
differs
substantially
from
everyday
image
captioning
in
both
visual
characteristics
and
description
conventions.
Specifically,
we
use
Fashion-Gen
[29],
a
large-scale
dataset
of
293,008
high-
resolution
studio
fashion
images
paired
with
paragraph-level
captions
authored
by
professional
stylists
covering
fine-grained
garment
attributes
such
as
fabric,
cut, fit, closures, and color. Unlike everyday captions that freely describe scenes,
spatial
layouts,
and
background
context,
fashion
captions
focus
on
the
design
specification
of
a
single
item,
making
the
domain
shift
explicit.
We
select
30
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
25
Zero-shot
caption
✗
“
a
prominent
star-shaped
patch
across
its
back
”
✗
“
body
facing
away
from
the
camera
. . . head
turned
to
look
back
over
its
shoul-
der
”
Grounded
base
caption
✓
“
a
large
white
patch
on
its
side
”
✓
“
standing
nearby
to
the
right
”
ReflectCAP
+
“
The
chicken
is
positioned
slightly
higher
than
the
goat’s
head
level
”
✗
Error
note
triggered:
Confirm
exact
placement
via
spatial
anchors
—
beyond
the
target
model’s
reliable
capability.
Fig. 8:
Limitation
case.
Error
notes
correct
zero-shot
hallucinations
(
red
→
green
),
but
the
extract-merge
step
introduces
a
new
spatial
error
(
blue
,
✗
)
when
following
a
missing-detail
note
that
exceeds
the
target
model’s
perceptual
competence.
exemplar
images
from
Fashion-Gen
for
the
offline
phase
and
analyze
how
the
resulting
notes
differ
from
those
constructed
on
everyday
images.
Comparison
of
Structured
Reflection
Notes.
Table
8
presents
the
Avoid
and
Include
notes
generated
by
the
same
pipeline
(GPT-4.1-mini
as
the
target
LVLM)
on
everyday
images
versus
fashion
images.
Without
any
modification
to
the
framework,
the
notes
shift
from
scene-level
guidance
(
e.g.
,
“Avoid
inferring
lighting
direction
or
time
of
day”)
to
garment-level
guidance
(
e.g.
,
“Do
not
add
clothing
fit
or
garment
length
details
not
clearly
visible”).
Notably,
the
fash-
ion
Include
notes
capture
domain-specific
conventions
that
have
no
counterpart
in
everyday
captioning,
such
as
interior
finishing
details
(
e.g.
,
lining,
surgeon’s
cuffs)
and
precise
pattern
or
fabric
texture
names.
This
demonstrates
that
the
offline
phase
automatically
distills
domain-adapted
Structured
Reflection
Notes
from
a
small
exemplar
set—once
30
images
with
domain-specific
reference
cap-
tions
are
provided,
no
further
manual
prompt
engineering
or
domain
expertise
is
required.
Qualitative Examples.
Figure 9 presents two representative examples compar-
ing zero-shot and ReflectCAP captions on AI-generated fashion illustrations, us-
ing the Structured Reflection Notes learned from real Fashion-Gen photographs.
In
both
cases,
the
zero-shot
baseline
produces
generic
visual
descriptions—
e.g.
,
“black
leather
jacket
with
multiple
zippers
and
buttons”
or
“tailored
black
suit
jacket
with
a
classic
lapel.”
In
contrast,
ReflectCAP
generates
domain-
appropriate
captions
with
garment
construction
vocabulary
(
e.g.
,
asymmetrical
front
zipper
closure,
notch
lapel,
welt
chest
pocket,
shoulder
epaulets),
precise
material
descriptions
(
e.g.
,
silver
zippers,
snap-button
details,
tonal
stitching),
and
accurate
fit
terminology
(
e.g.
,
structured
silhouette,
skinny
jeans
that
ta-
26
Min
et
al.
Table
8:
Structured
Reflection
Notes:
Scene-level
vs.
Product-level
caption-
ing.
Both note sets
are
generated by the same offline pipeline with GPT-4.1-mini. The
notes automatically adapt to domain-specific visual characteristics and description con-
ventions.
Scene
Level
Captioning
(DOCCI)
Product
Level
Captioning
(Fashion-Gen)
Avoid
Notes
N
avoid
Avoid
specifying
materials,
object
types,
or
inferred
object
roles
without
clear
visible
evidence.
Match exact color, pattern, and fabric
details
precisely
from
reference
or
vis-
ible
image.
Do
not
add
or
alter
visible
object
fea-
tures, colors, or counts without confir-
mation.
Do
not
add
clothing
fit,
style,
or
garment
length
details
not
explicitly
stated
or
clearly
visible.
Avoid inferring lighting direction, time
of
day,
or
environment
unless
clearly
visible.
Avoid
subjective
or
interpretive
de-
scriptions
not
supported
by
reference
or
image.
Do
not
add
unsupported
details
to
signs,
logos,
or
symbols.
Do
not
add
accessories,
footwear,
or
personal
attributes
unless
clearly
visi-
ble
or
mentioned.
Avoid
assumptions
about
scene
con-
text
or
climate
not
explicitly
shown.
Avoid
describing
visible
details
that
contradict
the
reference
(
e.g.
,
visible
buttons
when
concealed).
Include
Notes
N
include
Include
precise
object
condition,
ori-
entation,
posture,
and
spatial
posi-
tioning
as
visible.
Include
all
key
garment
design
fea-
tures
and
construction
details
(
e.g.
,
collars, closures, pockets, vents, cuffs).
Describe
all
visible
background
ele-
ments, environmental context, and rel-
evant
landscape
features.
Specify
distinctive
branding,
logos,
patches,
or
signature
elements
de-
scribed
in
the
reference.
Note
lighting
effects,
shadows,
reflec-
tions,
and
surface
details
accurately.
Mention
precise
color,
pattern
names,
fabric texture, tonal stitching, and fin-
ishing
details
exactly
as
given.
Mention
detailed
features
of
signs,
lo-
gos,
fonts,
and
text
including
illegible
or
small
elements.
Include
garment
fit
and
silhouette
terms
exactly
as
described
(
e.g.
,
shift
dress,
slim-fit
trousers).
Include
counts,
distribution,
distinc-
tive
markings,
and
compositional
or
stylistic
shot
details.
Add
all
notable
interior
and
finishing
details
(
e.g.
,
lining,
buttonholes,
sur-
geon’s
cuffs,
belt
details).
ReflectCAP:
Detailed
Image
Captioning
with
Reflective
Memory
27
per
to
the
ankles).
These
domain-specific
details
are
elicited
not
by
manual
prompt
engineering
but
by
the
automatically
generated
Structured
Reflection
Notes,
which
direct
the
model
to
attend
to
garment
construction
features
and
suppress
unsupported
fit
descriptions.
This
confirms
that
ReflectCAP’s
note-
guided
approach
generalizes
beyond
everyday
image
captioning
to
specialized
visual
domains.
G
Discussion
Perceptual
Boundary
and
Verification.
The
Include
Notes
in
ReflectCAP
guide the model to describe details it typically overlooks, but when this guidance
exceeds
the
model’s
visual
perception
capability,
it
can
instead
introduce
new
hallucinations. The current merging step adopts a conservative strategy that pri-
oritizes the base caption, yet it has limitations in fully filtering out hallucinations
introduced
from
the
detail
caption.
Moreover,
when
the
two
captions
conflict,
the
framework
is
designed
to
trust
the
base
caption,
but
the
base
caption
itself
is
not
guaranteed
to
be
always
accurate,
allowing
incorrect
descriptions
to
per-
sist
in
the
final
output.
If
an
external
verifier
or
visual
grounding
module
were
introduced
at
the
merging
stage
to
independently
verify
details
from
both
cap-
tions, it would become possible to aggressively expand coverage while preserving
factuality,
pushing
the
current
factuality–coverage
Pareto
frontier
further.
Domain-Specific Captioning.
As demonstrated in Appendix F, ReflectCAP’s
Structured Reflection Notes adapt to the fashion domain simply by replacing the
exemplar
set,
without
any
modification
to
the
framework.
However,
the
current
experiment
is
limited
to
qualitative
analysis
on
a
single
domain,
lacking
quan-
titative
evaluation.
If
validated
across
multiple
domains
such
as
medical
imag-
ing,
remote
sensing,
and
e-commerce
with
domain-specific
evaluation
protocols,
ReflectCAP
could
establish
itself
as
a
general-purpose
framework
that
can
be
immediately
deployed
to
diverse
specialized
domains
with
only
a
small
set
of
exemplars,
without
training
dedicated
captioning
models
for
each
domain.
Training Data Generation.
As shown in Section 5.1, fine-tuning with ReflectCAP-
generated
captions
maintains
factuality
while
improving
coverage
compared
to
human-authored captions. This is because ReflectCAP generates captions within
the
model’s
perceptual
boundary,
avoiding
forcing
details
the
model
cannot
ac-
tually
perceive
and
thus
suppressing
hallucination
amplification.
Scaling
this
property, ReflectCAP can be extended into a pipeline for generating high-quality
caption
data
for
T2I/T2V
training
without
human
annotation.
In
particular,
when
combined
with
the
domain-specific
adaptation
discussed
above,
this
could
simultaneously
address
the
scarcity
of
training
data
in
specialized
domains.
28
Min
et
al.
1)
Zero-shot
Caption
“The
image
shows
a
person
standing
with
their
arms
crossed.
They
are
wearing
a
black
leather
jacket
over
a
gray
hooded
sweatshirt.
The
jacket
has
multiple
zippers
and
buttons,
giving
it
a
rugged
look.
The
person
is
also
wearing
black,
ripped
skinny
jeans
and
black
shoes.”
2)
ReflectCAP
Caption
“The
image
shows
a
person
standing
with
their
arms
crossed,
dressed
in
a
black
leather
biker
jacket
with
a
classic
asymmetrical
front
zipper
closure
over
a
grey
hooded
sweatshirt
with
a
visible
hood
and
draw-
strings.
The
jacket
features
silver
zippers
,
a
wide
lapel
collar
with
snap-button
details
on
the
collar
points
,
shoulder
epaulets
with
snap
buttons
,
zippered
cuffs
,
tonal
stitching
throughout
,
and
a
flap
pocket
with
a
snap
button
.
There
are
multiple
zippered
pockets:
one
angled
chest
pocket
on
the
left
side
and
two
vertical
zippered
pockets
near
the
waist
on
each
side.
The
person
is
also
wearing
black,
distressed
skinny
jeans
with
rips
at
the
knees
and
some
frayed
areas
on
the
thighs.
The
jeans
taper
down
to
the
ankles,
where
they
meet
black
lace-up
boots
with
a
matte
finish
.”
Reflection
Notes
Applied
“Include
all
key
garment
design
features
and
construction
details
(collars,
clo-
sures,
pockets,
cuffs)”
“Mention
precise
color,
fabric
texture,
tonal
stitching,
and
finishing
details”
1)
Zero-shot
Caption
“The
image
shows
a
person
wearing
a
tailored
black
suit
jacket
over
a
white
button-up
shirt.
The
jacket
has
a
classic
lapel,
two
front
but-
tons
(of
which
the
top
one
is
buttoned),
and
flap
pockets
on
each
side.
Underneath
the
jacket,
the
person
is
wearing
an
additional
black
gar-
ment
that
resembles
a
wrap-around
skirt
or
apron
with
two
visible
buttons
in
the
front.”
2)
ReflectCAP
Caption
“The
image
depicts
a
person
wearing
a
formal
outfit
consisting
of
a
black tailored blazer with a classic
notch lapel collar
and a
two-button
closure
over
a
white
dress
shirt.
The
blazer
features
flap
pockets
on
both
sides
,
a
welt
chest
pocket
on
the
left
,
and
long
sleeves
with
no
visible
buttons
or
cuff
details.
The
blazer
has
a
smooth,
matte
finish
with
tonal
stitching
that
blends
seamlessly
with
the
fabric,
providing
a
tailored
but
not
overly
slim,
structured
silhouette
.
Underneath
the
blazer,
the
white
dress
shirt
has
a
pointed
collar
and
visible
white
buttons
down
the
front
placket,
buttoned
up
to
the
neck.
The
bottom
part
of
the
outfit
includes
a
unique
black
garment
resembling
a
wrap
or
apron
with
two
large
black
buttons
at
the
front,
creating
an
asym-
metrical
hemline.
The
fabric
appears
to
be
a
smooth
woven
material,
likely wool or a wool blend
, with no visible texture, pattern, branding,
logos,
patches,
or
signature
elements.”
Reflection
Notes
Applied
“Include
all
key
garment
design
features
and
construction
details
(collars,
clo-
sures,
pockets)”
“Do
not
add
clothing
fit,
style,
or
garment
length
details
not
explicitly
stated
or
clearly
visible”
“Mention
precise
fabric
texture,
tonal
stitching,
and
finishing
details”
Fig. 9:
Fashion
domain
qualitative
examples.
Zero-shot captions produce generic
descriptions
(
e.g.
,
“multiple
zippers
and
buttons,”
“a
classic
lapel”),
while
ReflectCAP
generates
domain-appropriate
captions
with
precise
garment
construction
vocabulary.
Green
denotes
fashion-specific
details
recovered
by
the
Structured
Reflection
Notes.
